yes, that is problem if mame still see both as mouse 1 then there is no way to disasble the other
What encoader are you using, does it support multi mice
also I would only enable it once, in mame, not in the UI, not sure how that works, but if it's set in mame it should work, when the UI launches it
the setting should be under general
scroll down to trackball X, hit enter, and spin the trackball on the X axis make sure it only registers mouse 1 x axis
Rince, Repete
hope this makes sense I stone cold sober, right now

Good luck! Meatbag!
BBR
oh yeah, if you mess up when trying to register an input move to a different input, then back to the one you messed up, hit enter then esc and it should register "none" and try again