Did you try reducing the sensitivity in MAME?
I haven't used the Ultimarc product, but I have messed around with some DIY setups.
They all worked except for the ones that used extremely low count encoders.
I got some cheap 12 or 16 count per revolution encoders from sparkfun that had detented positions which would have been cool.
It worked on a couple games, but on other games MAME wouldn't even detect any movement when trying to map it.
I swapped in a 900cpr encoder and it worked as expected. I've seen other DIY solutions using 360cpr encoders and they didn't mention any problems.
This was circa MAME v.142 in case there have been any changes since then.
I'm assuming the Ultimarc ones don't have a super low count, but you could try swapping in a higher count wheel if you can figure out the manufacturer.
Although not having the detents allows you to use the joystick as a spinner or steering wheel, I found that I really missed the feel of having the joystick lock into position.
My latest experiment is with 12 position mechanical switches which have enough of a detent to feel when you're moving positions.
I've stuck the parts together and turned the joystick to see how it feels, but haven't wired it up or used it to play yet.
Unfortunately the setup requires more depth than my CP has.