I guess I'm experiencing similar annoyance. Just two hours ago, someone called me stupid as well but in the language I understand. I do not like people calling me names in a language I do not understand; neither do I like being in a company of friends or colleagues and suddenly, they start speaking their own language. Knowing I do not understand it.
I think it is rude and disrespectful to speak or insult in a language to someone who does not understand.