Yeah...for me it is easy at the beginning as well. It is when I get talking more in depth and worrying about running out of stuff to say that I get nervous. I don't think I have the most serious anxiety, but I know what you mean.
I'm also a sophomore in college....don't worry, you are not the biggest loser ever. Today, I was doing the dishes "for fun" at the house I live in, and a housemate made a joke about "oh, I can't imagine what your Saturday nights are like! they must be wild...." Usually she is nice, and I don't think she thought I'd take her seriously.
I know it's hard, but the less you call yourself a loser the better things can get. Even just eliminate the word, use something that's actually descriptive. I don't know, I'm okay being a loner, but I try not to call myself a loser.
Also, I used to go the library on weekends...it made me feel better to see other people alone who weren't "going out"....