I live in an apartment building FULL of parents and their wittle adowable babies in giant carriages and toddlers dragging toys along the floor and I constantly get stuck in the elevator with them. I HATE IT.
You have to make "oh, hi there! oh, you're CUTE!" talk to the baby, otherwise the parents think you're crazy. And the toddlers STARE at you, probably because they're used to attention. This is horrible because I don't feel comfortable being watched as I stand there (trying not to slump or have a bad expression on my face) knowing this stupid little kid wants me to pay attention to it and its mother wants me to say how sweet and cute it is. So I end up giving a really fake smile, saying a really pathetic "hi there" to the kid, and then trying to look busy with whatever I can until I'm out of there. This usually results in the kid giving me a look because I didn't seem "happy" to them (seriously).
On the other hand, I did two summers as a volunteer counselor for early childhood camp (roughly 2-5) and I only had problems with those kids if I was supposed to discipline them (can't do it). Same with my paid counselor job for 7-9 year olds a few summers ago. Maybe because camp kids are trained to want to like you until they decide otherwise?