I hear you. An unfortunate byproduct of depression and anxiety is that sometimes people become so focused on their own struggles that they fail to see the struggles of others. People who focus purely on their own wants and needs come across as selfish, even if they don't intend to be.
Personally, I hate to be a burden on my friends. So when my depression and anxiety started coming back I'd just not hang with them when I was really down cause I didn't want to come across as selfish. Problem is though, it of course led to even more isolation which made it worse.