pbj gotta pbj...
But in all seriousness, I actually agree with him. Unless trackball games are just as important to you as other games, then leave it off.
Either build a panel with it fully integrated like it's supposed to be there, or just leave it out entirely. Quite frankly, Marble madness and golden tee require some very aggressive play. Hanging a trackball off your existing control panel will probably not only look ugly, but will not withstand even the most moderate abuse. Furthermore, some surface area around the track ball is necessary for comfortable play and that negates the desire to attach something small.
Maybe I'm missing something here, but this sounds about as strange as having a removable flight joystick, steering wheel, or guns.
Either build your control panel like it was supposed to be there the entire time, or just leave it out.
When you want to play the 3 games you like that use the trackball, then use pbj's suggestion.