I've created a poll unit. I initially added categories but we determined we weren't going to have categories. I've tried to remove them but am unable to.
if you own a premium package you can delete all voters from your contest and then you can remove the categories.
Otherwise you would have to create a new contest.
What if mine hasn't started, therefor there are no voters yet? Can I remove the categories before the contest begins?
Yes, before there are any voters, you should be able to remove / change the categories.