Editing Voting Categories after publish

I need to remove the voting categories but in the editing stage it will not allow me to change or remove them.

You can not remove voting categories when there are already voters. If you delete all voters you can change or delete the voting categories.