Plenty of people wore one at the pool based one I did last year. Some for comedy effect, some for that extra 0.05% speed. Lap counting wasn't an issue as it was up and down in one lane, move to the next lane, up and down that lane, move to the next and so on zig zagging up and down the pool was basically the way the laps were counted.
If you have a piratey one, I say wear it!