It could be mame (either ctrlr file or cfg file), or the encoder you're using and it's mapping. If you're not using a ctrlr file and the in game "Input (general)" doesn't have that key combo as an input for UI Cancel, it's the encoder; as ctozzi said, change that ASAP.
I hate, hate, HATE that combo exiting mame; in my case it was (pre-ctrlr files) mame's default mapping for hotrod CP (which has been changed, BTW), and ever since, IMO UI Cancel (aka "esc") always should have an "exit only" button used for nothing but exiting mame (I prefer a combo actually; a "shift" only button + that exit only button). Others feel that a non-extra button combo is okay, though, so set it up whatever you think is okay.