Analog joysticks pots can wear and have bad-spots. Especially true with a very old or heavily used device.
Besides the software end, such as deleting older config, ini, save-state, (etc) files...
Replace the pots with linear tape, not audio taper, pots.
Calibrate the stick in Windows first. Check for fluid precise control.
Check precision on PC games, or emulators, other than mame.
Test other analog joysticks on mame. Try other USB ports.
Make sure the pots are the correct ohms for the PC or encoder. Reflow solder connections as well.
(if wire is too thin, could be some broken intermittent connections as well)
Some games have auto-calibration, like starwars, where you have to roll the stick around the edges a few times.
Some have calibration menus. Research how to get into them. Some are confusing and non-intuitive.