I get a lot of anxiety when I have to speak French too, so don't worry you're not alone! I find that the more you practice and the more you study, the more confidence you'll get in yourself. As an ESL teacher, I've encountered a lot of students who got anxiety or were too shy to speak English in front of me, due to the fact that I was a native English speaker. I would always try to get them to relax and remind them that I was there to teach them, and not to judge them. The more you speak a language, the better you get at it. It takes time, and there are some rude people who will simply be rude about it, try to ignore them and continue your quest for knowledge!