I think any time a person is ignoring someone, or giving them the cold shoulder, it will always appear rude. People with SA don't want to ignore people, but its the nature of the disorder; to avoid and hide. Until people with SA can eliminate this uneasiness around others, they will always be the target of misinterpretation.
The problem is that people with SA need to have some good rapport and emotional connections with others, yet this nature of the disorder doesn't allow that. I would assume that if SA'ers had one good, emotionally mature person to truly support them and bond with them on a deep level, that an emotional growth would occur in such a way as to pretty much cure the social phobia.
Child-like emotions and/or damaged emotions are the root of the problem. Thus emotional maturity and healing are the solution, which can only come from someone who deeply loves and cares for someone with this issue.