I think there is no problem with it if it helps, and I have thought about doing the smae myself.
And I am fed up to the back teeth with people trying to read me, and then misintepreting my anxiety, calling me names, and making fun of it.
So you think it's rude to wear sun glasses whilst talking to people? Well it's even ruder to call people names, to judge them, to make fun of them, because they've read their anxiety and got it all wrong.
People who don't like talking to me with sunglasses? Get over yourselves and I probably wouldn't choose to talk to those people anyway.