I rarely wear mine because I have to take it off for sport and it ends up being easier/less likely to misplace if just put away in a safe place. In the end I bought a really cheap wedding band which has no sentimental value so no worries if I lose it, but to be honest the hassle factor is the same and I still don't always wear it - but I never have that heart stopping moment thinking I've lost it. I think I caused a bit of bemusement when I went into Argos and asked for their cheapest wedding rings and walked out saying that ??30 was more than I wanted to spend.
I don't really see a problem either way - lots of reasons not to wear one (for me, my fingers swell a bit when I run), but equally no reason to remove it if there's no problems. What makes you think you should or can't remove it?