Part of relationships is taking your partner's values into consideration.
So... if family is very important to him; as It probably is-- then it would look best on you to make an effort even though it's awkward and painful for you to be there when you don't speak the language.
You don't have to, of course... but I'm sure he would appreciate the extra effort it takes for you to do such things.
It could only strengthen your bond for you to get to know his family.
I think you should go, it's not the people who gathers there that are the center of attention, but the niece. If there actually would be negative thoughts about you in the family, then you would improve the relation by at least showing up, maybe even engage in some small-talk. If you're not even there they can start making ideas about you that aren't true. If you feel that the problem is the form of the relationship I think they're having that talk with their son and not you. Maybe your apperance will make the niece happy.