Eye contact's also a problem with me. When someone looks at me, their eyes seem so incisive and piercing. It's like they're staring through my eyes and into my soul, and I can't help but sense that they can tell if I'm uncomfortable...which of course, makes me even more uncomfortable. When I'm talking to someone, I'll alternate between bowing my head so that I'm sure of what they're saying (especially in loud places) and looking up to make eye contact.
And yeah, I agree with MotherWolff. Western culture is such that if someone doesn't make eye contact, people would tend to assume this person has something to hide...that is, if they don't assume they're being disrespected by this lack of eye contact.