On the game PCB there is a dip switch setting for mini or large trackball. So if you are using a smaller than 4.5" trackball you may need to adjust the sensitivity of the trackball/mouse or in MAME set the dip switch to mini. I think it will basically be something for you to play with to get the right feel.
Based on my calculations, and a 4.5" ball (which I believe is the correct size), one would need to set the sensitivity of a 3" ball to be 1.5x (or 150%). That is, of course, if the MAME code is correct at 100%, which I cannot confirm, when used with a trackball which uses quadrature decoding. IIRC, a number of early Atari titles didn't, so knowing how the original handled the data from the control is an important first step in matching the actual sensitivity value.
But
if the original
did use full quadrature decoding, things would get a bit worse. In this circumstance, the issue with doing things the way I described above is that, much like using a low-res spinner and increasing it's sensitivity for a game like Arkanoid, you would be upscaling and therefore lose accuracy as a result. In other words, you can't add resolution, you can only even out the amount of movement at the control to match the same distance covered by the cursor on the screen. However, if this turned out to be the case, you can increase the resolution of the trackball by replacing the standard encoder wheels with a higher resolution variety, and adjust the sensitivity to match what you need it to be. It's one of the benefits of using high resolution encoders, such as the ones we offer with our trackballs.