Recording out of mame doesn't work too well, use a third party program like jakobud suggested....
These are the settings you want:
Turn off hwstretch, direct3d and direct draw
Set window mode to 1 and max to 0
Now the game will pop up on your desktop in it's native resolution without "blending" issues.
Select the window with your capturing program and your all set.
For capturing settings I suggest a modest 10fps with the sound at 22 khz or less. These are the settings peter and myself use.