I can totally empathise with this.
Some of my old work colleagues had a tendency to do this (particularly one). I found it rude. Like it's already been said however, people don't realise that they're doing it. We tend to notice as we're so conscious of our interactions with people.