I dont think it has something to do with your overall appearance. I have a gut feeling that you have the 'sanpaku' eyes (just google it) which people might find piercing to look at.
Believe me, when you are ugly, people will tell you directly.
Well, either directly or indirectly. Depends on how confrontational the person/people are. I can recall only one time sby i knew said i was ugly & also said to wear a paper bag over my head & to make ghost noises (which i did right away, to call their bluff). Another time the person said (after not seeing for a year/so) 'still as ugly as ever'. I can recall one time where sby said loudly 'god he's ugly' (direct comment, but said indirectly). The rest of the times, were indirect stuff, like being laughed at by groups, being called names, belittling, bullying, etc.Believe me, when you are ugly, people will tell you directly