The easiest way is to physically wire the two trackball buttons into two of the player buttons, instead of running those to your encoder.
Then all you have to do is set MAME to use those instead of it's default keys, and they will work perfectly.
I would recommend replacing buttons 5 & 6 in the bank of buttons, so that there is minimal impact on everything else.