Well, I'm just one year shy of finishing as an undergrad, and I can say that college is definitely possible as a person with SA!
But, it also depends on what you are expecting, as vj said. For most people, the "college experience" includes a lot of social contact. If that's what you want out of your college experience, you'll have to work harder than most at it.
I myself have stuck to the classes, which aren't that bad. I'm at a large university and classes are usually 100+, so I don't get the same sense that I'm under a magnifying glass that I got consistently during high school. However, there are plenty of cases I've had to work in small groups regardless of class size. These times have been the most hectic for me, but I worked through it.
And how daydream insinuated, many classes can now be taken online, or at least large portions of them are available online.