With freedom comes responsibility. You are not a responsible user of your freedoms when you use them to debase other people.  Moreover, defending yourself against criticism for your actions by wrapping yourself around your individual rights to freedom shows a great lack of responsibility.

Not to mention that it shows that you are incredibly thin-skinned.