Yeah, but the rollers will still be PC trackball rollers.
It is not the case that slows it down, but the bearings/rollers that the ball rolls on.
I guess it would be a low cost solution anyway, so if it doesn't work out all that well, you wouldn't be down all that much money, and an arcade trackball could then still be an option.
I am curious to hear how it works out for you, if you choose to do it.