Mock it up. That's really about the best thing to do, go track down some cardboard and make a mockup to see if you'll be hitting your hands on stuff or whatnot.
I had to pull the trackball out of my bar-top design after the mockup, because while it'd physically fit, playing anything other than centipede wouldn't be possible.
If you were looking primarily for a windows mouse, and you wanted something small, take a look at the Logitech Marble Mouse. I was planning on hacking one for the bartop project. It's a 1.5" trackball, and after I took mine apart, it looked like it'd be manageable to get it working in an arcade CP. Happs also makes a 1.5", but it runs around $100 instead of the $20 for the marble mouse.