I'm guessing, but...
Sounds like the UI Cancel key (aka esc, aka UI escape) is a left or right shift, ctrl, or alt key, and the pause button is the other shift, ctrl, or alt.
Don't do this (for win mame, at least). Windows treats these keys differently when mame is paused than when it isn't. When paused, windows sends mame both left and right of these keys when either left or right is pressed. So if pause is left shift (for example) and UI Cancel is right shift, you'll get what's happening to you.
Just use a different Xarcade key for pause or UI Cancel.