What you're seeing has been a common issue. The way mame is set up is that joystickk data overrides mouse data (assuming both are mapped to the same input). This isn't much of an issue for 4/8-way digital sticks. But with analog joysticks with no deadzone, if the joystick is just off from center, mame uses the joystick data instead of the mouse data.
So there are a bunch of ways to fix this. They include:
a) disable joystick in mouse input games
b) remap the inputs so the mouse is mapped but NOT the joystick
c) increase the deadzone settings in mame
d) remap the u360 so it has a bigger deadzone
I prefer b), which seems you've tried. Hmm, are you remapping in "Input (general)" or "Input (this game)"? Which input names are you remapping? (should be mapping mouse to TB/dail analog, and unmapping the joystick from the same + TB/dial inc/dec)
For testing reasons, disable joysticks and see if it work and if you can remap the mouse. And enable verbose mode so you'll get more info displayed at the command line.
If the mouse works for, say, arkanoid, but you still can't remap, you might be trying to remap to a digital port instead of the analog one. If it doesn't work in mame but does in windows, you could have multiple mice installed (IIRC that was when you were forced to use each mouse seperately), or have mouse disabled in sub ini files (usually in the ini folder). If it does work in mame and remaps, remap with joysticks disabled in Input (general so you do it only once, and then reable joysticks.