This is one of the downfalls of the Happs style microswitch assemblies.
The button is a wide hollow tube... and so it acts like a speaker, and amplifies the sounds. The thin walls, do not insulate the sounds well.
The plunger has a lot of wiggle room... and so it often slaps into the sides. The button is designed to bottom out... so the impact of the hit,
will also resonate and be amplified.
Compare this to a pinball or real 80s era leafswitch button + assembly. These have very narrow shafts, with thicker walled assemblies, and use
harder plastics. Due to the plunger shaft being in a smaller diameter assembly... it doesnt rattle around much. Also, because they used a longer
button travel, and true arcade leaf blade switches.. it was far more rare that they were bottomed out.
You might be able to reduce the sound, by using a dampening material... such as a rubber coating in the button hole, under the button lip, and under the plunger... to absorb the impact and vibration effects. Also, you might be able to fill the very center of the button with something... to keep the air volume and sound-space far more limited in its wide open scope.