Take a look at this PDF download entitled "12 Things You didn't know You could do with MAME":
http://freepdfdb.com/pdf/12-things-you-didnt-know-you-could-do-with-mame-630337.htmlOR
mamedev.org/misc/cax2009.pdf
On one of the latter pages it talks specifically about how to record games with MAME. Part of it is copied and pasted below.
MAME also supports recording in several formats
•To record an AVI/MNG/WAV:
mame <game> –aviwrite <filename>.avi
mame <game> -mngwrite <filename>.mng
mame <game> -wavwrite <filename>.wav
•To take an in-game snapshot press F12 (configurable)
You can control the size of snapshots and movies via the –
snapsize and –snapview options.
•By default, it is the native screen size
Once you get a proper video created, you simply load it up into Mala by pointing Mala to it as a startup movie.
GUI>>Start.
Your movie MUST be in a format and codec that Mala recognizes or it won't play.
Not sure about your Alt-F11 question, but I'd refer you to this site for better explanations on MAME special keys/functions:
http://strategywiki.org/wiki/MAME/Special_KeysBest of luck.