make sure to look at the Ohms rating on the speakers vs the amp. If they are too different then something will get hot (in the case of car speakers with a home amp, the speakers may be the things to get hot). I have never had a problem with this personally, but just a warning. Basically, don't crank the volume all the way up.
The reverse is true with using home speakers in a car, you can blow your car stereo that way as the amp in it will get too hot due to the higher resistance of the home speakers.
You are on the right track but you have things backwards. The higher the impedance of the speakers, the cooler the amplifier will run, because it is making less power. There is never a problem with using higher impedance speakers than an amp is rated for; in fact, it provides the amp with a nice and easy life. The downside is that your amp makes less power.
For example, a typical unregulated amplifer that is rated at 100W with a 4Ω speaker (typical car speaker) will only deliver 50W with an 8Ω speaker (typical home speaker). However, that same amplifer will deliver 200W with a 2Ω speaker (assuming there are no protection circuits to prevent it), which could be too much current for the components of the amp and cook it.
However, there are variables that tend to work in people's favor, preventing them from cooking their amp when using lower impedance speakers than the amp is rated for. For starters, music is dynamic, and with changing frequencies, you get changing loads; often substantially higher than the speaker's nominal impedance. Also, source units and amps tend to have volume/gain controls; which can be viewed as power controls. For example, an amplifier that "goes to 11", and makes 100W at "11", will only be making a fraction of that power at less than "11".
So, if you take an amp that is rated stable for 100W @ 4Ω, connect it to a big 2Ω resistor, feed it a 1 kHz tone, and crank everything to the max; you will probably cook the amp. However, if you hook it to a 2Ω speaker and play music or other dynamic forms of audio (as opposed to constant frequency test tones) at a reasonable volume, you probably won't harm anything.