I know back when I was on the high school's football team I was often alienated because of this, essentially. It was pretty awful. To this day I can't stop asking myself why I ever joined in the first place. Long story, but basically, in that instance, they did treat me differently for it.
Besides that (and after my last year on that team), however, everywhere else I go I'm just mistaken as this "cool/chill/relaxed/laid-back" kind of person a lot of the time because of the social phobia and not being so extroverted. Maybe a bit shy, but SA doesn't stand out to them for some reason.
In your case, I don't mean to say any of this is your fault or anything, but do you say hi or bye to him? Again I'm not saying you're in the wrong if you don't do that; but simply that if one person isn't greeted first, they're not comfortable greeting the other. (Which anyone with social anxiety could attest to, it's easier to greet someone who's greeted you before.) As far as everything else, he's probably simply generalizing; if you don't look approachable then he/she may not think you are, whether you really are or not. Besides the last paragraph, he/she doesn't strike me as "rude" rather than "mistaken". In the last paragraph, it seems rather odd that he/she would just bug you like that.