Should I change the 'mouse speed' in Windows?
No, it won't effect mame (assuming you are using XP or Vista).
I've tried changing the 'digital speed' and 'sensitivity' I believe in MAME, and now I think the trackball is too sensitive.
You don't need digital speed, it's if you were using digital inputs, such as buttons instead of a real analog device.
So set sensitivity to your likings. Since their are a dozen TBs and spinners, and hundreads of games that used TBs or spinners, and thousands of PC mice, it's impossible to have one fixed sensitivity value that will work for everyone on every game. FYI, the default sensitivity value for each game is usually set to the best value that works for the developer of the game's driver in mame, and most of them have a standard PC mouse AFAIK.
The main use of the trackball is Golden Tee for me, and while I don't want people to have to get a running start and smash their hands into the monitor, I also don't want to take the skill out of it by having it too easy to hit long drives. There seems to be this give and take with the other TB games, like Shuffleshot; if I get the settings right for one, the other is way off.
One game's sensitivity shouldn't effect another game's, unless you try to set them to the same number. It doesn't work that way. You'll have to set each game to it's own sensitivity value (especially if the arcade hardware was different).