I have a similar fear on social media and it relates to specific people looking through my posts and adding me - coworkers who see only my professional side, a friend-turned-casual dalliance-turned-friend again from over a year ago who has an urge to comment on everything I post, and needy relatives who post gushing long-winded comments. I feel obligated to add these people because I have history with them or I'm forced to interact with them often. Maybe another part of it is I don't like being too accessible.
I remember Facebook used to let you disable anyone from friend requesting you, so the initiation of Facebook friendship is solely on you. If they would let you do that, I would re-activate my account - I'm afraid of the reaction I'd get from a person mentioned in the list above if they figured out I blocked them.
I tried to get into Instagram but I don't get the point, and I believe I may be slightly outside of the targeted audience's age bracket because I don't know anyone my age who uses it. I get follow requests from camgirls and don't understand why, because all my pictures are of trees or cats.