I think there were planned (and abandoned) projects along these lines. I recall ArcadeOS, an interface possibly considering this route, but AFAIK that's a dead project.
Linux builds are your best ticket; I'm not sure how hard it'd be to compile MAME on QNX, but that'd be pretty amazing. QNX is tiny and very featurr-rich, though it never really went anywhere in standard use; it can even boot, with a full GUI and apps off a 3.5" diskette... Yeah, tiny footprint and it'd make an ideal foundation for a MAME-OS, though I don' remember either how much POSIX-friendly it is at this point, or even how much of the MAME and MAMEUI source is POSIX-compliant. Certainly, it'd be neat to code a small-ish interface/GUI for MAME on QNX though, as it'd be a good way to recycle old tiny SD or CF cards. F.Y.I., CF cards are simply PATA devices--normal 2.5" hard drive style with a slightly different pin configuration--and you can drop them into any system as a boot device, so you might imagine the possibilities of such a small OS for instant-on MAME machines.

DOS is not the way to go due to limitations in interfaces and device support. BackTrack5 is an excellent Linux distro and it supports flashcard and DVD booting builds, so you may want to investigate that as well; I use BT5 for a second OS on my Mini-9.
You should look into distros & builds for your hardware setup that are small, will load from USB storage and support data writing on the boot medium for optimal use. That's one limitation of BT5: You can't write to the boot device at all if that device is a flash card, although you can always have MAME on a second device. I keep my MAME setup and ROM files, etc. on an 8GB flash card for portability and reliability reasons.
Really, using CDs or DVDs isn't advisable as your game settings will be lost on exiting the game and you'll need to re-configure it each time. -GG
Edit: PhMAME is a QNX port of MAME, but it's outdated at 0.60x... It may npt be too hard to update itto the latest source, but then again, I 'm not very familiar with the ins & outs of the MAME source, so I'm not certain about that. I may have to install QNX 6.1+ and PhMAME just to try it out for kicks and to see how hard it'd be to make a tiny bootstrapping version at its current build and go from there.