Such situations can be unfortunate.
There are people on OCD forums who fear they might be molesters or such themselves.. It can be a very difficult situation..
Have you tried talking to your family about this?
If you are sure you have no tendencies toward this, just try to be as normal as possible, and maybe when you get a more social life or a steady girlfriend, all this will be forgotten?
Maybe just keep a little distance from kids on such occasions and make sure anything you do is 'appropriate'..? (It can be difficult with kids, I know... Teachers have to be careful with such things too..)
Maybe there is a relative with some 'issues' in the family and they're just not telling you about it? (Families can have secrets that can really influence lives, even when nothing is said about them...) Or people have own problems and then extrapolate on others?
I think if the kiddie came to you, you are apparently a likeable person, so that is something to build on. Parents or other people can have all sorts of irrational beliefs. Sometimes it's enough to talk about stuff or just focus on other things... (and prove them with time that they were wrong)