OP, be sure to read the last part!
Mame/mame32/mameUI have an in-game menu that you get to by pressing tab key (the default). This is what the xarcade instructions are talking about.
The three all can directly start a game.
Mame/mame32/mameUI all can start without starting a game. This is a "Front End" (aka FE) view. Mame's is very minimal.
Mame32/mameUI FE has a drop down menu to do things like change mame's settings. You can do the same things by hand in mame, but the menus in mame32/mameUI mouse-a-ize it. However, there is no way to directly remap inputs as in xarcade's instructions.
Mame has a ctrlr file in the ctrlr folder for the Xarcade. Mame32/mameUI doesn't come with it anymore; but you can download the same version mame and use that ctrlr file. If copied into the mame32/mameUI ctrlr folder, it will show in the drop down menu wbassett mentioned. But this has nothing to do with the TB.
However, sounds like you are using mgalaxy FE, so you aren't going to the mame32 FE, so you don't have that drop down menu. But since you're using a different FE, just set up the ctrlr file through that FE.
There is no such thing as a dos mame for a long time. Official mame is a windows application. Don't confuse window's commandline with dos.
Back to the OP first question:
The main thing I would like to do besides using it in a game is use the trackball to scroll through my mame list on my frontend.
This is a FE issue, and has
nothing to do with mame. Set up your FE to use the mouse/TB. If you can use the TB depends on your FE, and I don't know mgalaxy.