I also, can relate to this situation, but don't let the fact that you didn't have panic attacks make you conclude that you don't suffer agoraphobia. Certain situation like even leaving the house trigger you not wanting to go the horrible road of going out there, and thus the chance to encounter a fullblown panic attack is nihil, personally I'm in the same boat for over 10 years now, countless of times I was that close from having a panic attack while only putting my shoes on and grasping for my coat, that only reverting back is all that was really possible.
And while the symptoms really point to agoraphobia, in my case, and pretty much all cases of fullblown agora are only a result of even deeper fears, being social phobia in the majority. I can relate to the fact that you have a hard time trusting anyone who wants to dig deeper in the root of your fears, because it's a normal thinking pattern that this will ultimatelyu lead to forcing you to do things you currently see as "way too much over your head" as to where you stand right now.
Indeed, small small steps, and once in a while forcing yourself a little, but once you feel you have to force yourself any single time, it's perhaps time, to take a small step back, and try again, and the hope and knowledge that you'll progress forward along the whole long way.
