This has come up before... along the same lines as why women are comfortable about wearing a skimpy bikini yet scream if someone catches them in their bra and pants...
It's the context - I'll wear a bikini at a beach or swimming pool, but not at the shops.
If I was at home in my underwear, I would put some clothes on if someone knocked at the door.
If one of my male friends saw me in my underwear I wouldn't scream (nor would I if it were a stranger), but I would probably feel a bit uncomfortable because underwear is normally kept out of sight.
Most blokes wouldn't walk around the shops in their swimming trunks, and I think most would put some clothes on to answer the front door rather than do so in just their underpants.
Edited: 26/07/2012 at 14:12