The rule-of-thumb is the higher the impedance of the speaker, the higher the quality of sound. But higher impedance speakers need more amplifier wattage to get the same volume level. A higher impedance speaker will never hurt an amplifier designed for a lower impedance speaker. It's just that the volume won't be as loud.
The lower the speaker impedance, the greater the distortion in the sound. If you use an amplifier designed for 8 ohm speakers and connect 4 ohm speakers, the lower impedance will draw twice the current from the amplifier and at a greater distortion too.