It would be easy for me to read the 'whale' commentary you gave and think 'well you're ugly on the inside, mystery solved.' But I imagine you're more likely frustrated at the whole situation and venting (in the wrong direction. I mean come on, aren't you sick of that behaviour when it's directed toward you? Judgements based on physical appearance, criticisms of your physical appearance? You clearly are). I'm not trying to scold you, just making you conscious of that unbecoming way of thinking about others...
I think your problem is more to do with how 'approachable' you are, as others have stated. You'll either have to have a frank discussion with yourself about that or ask those close to you to be very honest with you about the way you come off.