You can do it with autohotkey.
Here's how sevredhed explained how to do it:
at the bottom of this post is the content of the ahk file. the key to getting this to work is that the button you are pressing has to be mapped to a different key than both your pause and exit keys. for instance, i use the default mame keys for pause (p) and exit (esc) i tried it with my button mapped to the P key on my encoder, but it did not work properly. i found that in order for it to work, i changed my button to be mapped to the PgDn key, which was unused for anything else. This way if i press PgDn quickly, it sends "p" or if i hold it for around 3 seconds, it sends "esc". this works very well.
no in order for this to function, mame has to be compiled to support direct input...by default, the newer versions do not. There are tutorials on how to do that around here somewhere if you don't know how. (i always compile my own anyway because i wanted the hi score, no nag patch) if you need help with this, let me know and I'll find the instructions i used.
here is the script. you can redefine the keys as you see fit. I cannot take credit for the script, it was created by forum user WXFORECASTER. everything below this line needs saved as an .ahk file:
/*
AUTHOR: WXFORECASTER 10/12/10
DESCRIPTION: DESIGN OVERRIDE SUCH THAT A SINGLE KEY PRESS AND KEY HOLD
GENERATE DIFFERENT FUNCTIONALITY
*/
/*
BEGIN HOLDESCAPE CODE
SET USER DEFINED VARIABLES BELOW
*/
timeToExit := 1.6 ;time in seconds key must be pressed before sending exit (this can be a decimal e.g. 1.5)
emulateKey := "PgUp" ;the key we are pressing/emulating
pauseKey :="p" ;your MAME pause key
exitKey := "esc" ;your MAME exit key
exitScript := 0 ;0= leave script running, 1= close script when the MAME exit key is fired,
;Make it so the following code only works if MAME is the active window
;#IfWinActive ahk_class MAME
;Establish Hotkey Mapping based on the emulate key above and trigger a chunk of code below (lblPauseExit)
Hotkey, $%emulateKey%, lblPauseExit
return
lblPauseExit:
KeyWait %emulateKey%, T%timeToExit%
if ErrorLevel = 1 ;timeout was reached...exit MAME
{
send {Blind} {%exitKey% downtemp} ;Send MAME Exit
send {Blind} {%exitKey% up}
if (exitScript = 1)
ExitApp
}
else
{
send {Blind} {%pauseKey% downtemp} ;Send MAME pause/unpause
send {Blind} {%pauseKey% up}
}
return