It's because they define us by our quietness. To many more social people we seem weak, small, and awkward. Society views us as outcasts and loners, even if that's not who we are on the inside. I can't tell you the amount of people I've met that treat an introverted nature in itself like a problem. How many times have you been asked, "Why are you so quiet?" It's as if loud and outgoing is considered the norm, and anyone who differs from that has a problem. This is why we feel pressured to make friendly conversation with strangers even though it's uncomfortable. It's what the rest of the world considers acceptable. People with SA aren't sympathized with because others think we all want to be isolated and don't want people to talk to us because of how we behave. That may be true for some but certainly not all of us. It's like others think we bring the condition upon ourselves and should just start talking more.
Not to say that we are the only misunderstood group of people. Just as we have all been told our social fears are irrational and we only need to talk more, people with depression are told to just cheer up. People with anorexia are told to just eat. People with ADHD are told to just sit still and pay attention. The list goes on and on, but basically, many mental disorders are misunderstood because they are like the extreme version of a feeling that most people feel. What is not big deal in their eyes consumes us. They probably try to sympathize, but think we feel nervous like they do when they make a presentation in front of a group and they have overcome it. So in their mind, we should be able to as well.
Not to say that I understand or know how people think. Or that I think this applies to all people without SA. This is just my assumption based on things people have said to me and others people.