I don't know that you have to let go of that fantasy. Of course, it's not be good to be fixated on that fantasy, but just stop thinking of it as a fantasy! Think of it as a goal.
Maybe the reason you're feeling so down about it is because you're not doing anything to achieve that goal? What if you tried going out and making just one friend, or just one acquaintance, or just try talking to a shop assistant and smiling at them! Actually doing something about it is much MUCH harder than sitting at home wishing you were better and had lots of friends, but if you really want it, then you can do it.
Just imagine if your fantasy could come true. It can! It's not a ridiculous fantasy, it's an achievable goal!
Would there be any night classes near you you could start attending? They would be really good, there would be no pressure to make friends because you'd be there for the class, it would be a relaxed, friendly situation and you could ease your way into interacting with people.
I really really do hope you do something about it, it doesn't have to be a fantasy! If you work at it, you can have it. If you're sure you have APD, go to your doctor and ask him/her to refer you to a psychologist or psychiatrist and then talk to them.
Good luck!