OK, first off. the reason there are mounting holes in the motherboard is because it is designed to be put in a case.
That's because most sane people do this, mostly because they don't want exposed PCBs lying around. This has nothing to do with cooling however. There are several components that should be cooled regardless if it is in a case or not. (cpu, video, etc.) If these components are cooled (water, heatsink fan combo) it matters not if they are in a case. There should be active cooling and enough ventilation to keep the ambient temperature to a reasonable level. (which may, incidentally, be achived by taking the computer out of the case under certain circumstances)
That being said, the only component on the motherboard that I know of that has an even remote chance of thermal failure is the northbridge of certain chipsets. It should be noted that most of these only have a passive heatsink on them to begin with and are not a signifigant problem unless you are overclocking (and if you are, the first thing (well second, after adding a pound of copper to the top of your CPU) is to add an active cooler to the chip ~$20 including self-adhesive pad.
So, cases (aside from the form factor) have no effect on motherboard design. The reason that major PC manufacturers constantly shout "case must be closed" is because they use plastic baffles to direct airflow and reduce the number/quality of fans in their cases.
*whew*
-C