Hell, at least men with mancaves have seized back some small portion of their homes for their own use.
I'm with Le Chuck on this one. Anyone who has been married for a long time understands how women will take over the house, declaring each and every room for themselves (or for the kids), not leaving any space for the man. Usually they end up "conceding" to give the man a space in the garage or a dark and musty basement - both places they won't go and no guests will inadvertently see. Hence the term "cave". It ends up being a place men can be comfortable in. Not a formal living room without a television (wtf), not a dining room, kitchen, clean bathroom, or bedroom, but a LIVING space that has everything a person who doesn't care about fashion, décor, or impressing the neighbor ladies would want in a room. Usually the women of the house are ashamed of it.
At the same time, however, the term gets overused. If you are simply converting a room into an arcade, it isn't a "man cave", it's just a game room. If there is a lot of natural light in the space, it certainly isn't a cave, and if it's in any of the main living space, it definitely isn't "man cave" material. Hell, if it is decorated with style, it isn't a man cave.
It took me several years to convert my home back to what I wanted after my divorce. For nearly 20 years I had been conditioned to just expect to make the house the way SHE wanted it, and at first any changes I was making were difficult to make without input from a woman. Then I got in the swing of it and started redecorating for ME. I don't need a man cave because the whole house is mine to do what I please. I built a 300 sq ft game room out of the back porch. I converted the basement family room into a theater room. And the rest of the house is set up the way I like it, even if some women walk in and shake their head. (Actually aside from comments like "masculine", most women are impressed with my house, and quite surprised that a man would be capable of designing, building, and decorating rooms the way I did. Of course if any of them ever moved in with me they would be changing things, lol)