Good for you. If you can get through life without punching at all, more power to you. Yet ... If a people are oppressed, and all legal / non-violent means have proven ineffective, what can they do?
The vow of non-violence is something a person can take only for themselves, after much reflection. It's not something to be recommended to or imposed on others.
I've talked my way out of many potentially dangerous situations -- including a few attempted rapes and murders. I believe that God gave me that ability in response to my (personal) commitment to non-violence. But I would not expect anyone to follow this example. Each person, and each people, must decide for themselves -- and only for themselves --how much they will take and what they will do about it when enough is enough.