... If you're trying to pass the LeMans encoder wheel (big) through the mouse's optics (really tiny), they may be incompatible. In order for an encoder wheel to work with a set of optics, the spacing between teeth has to correlate with the spacing between the optics just right, or the optics won't properly register every tooth, or even get the direction the wheel is spinning right.
For best results, if tooth width = gap width = X, and distance between sensors = Y:
Y = X * (n + 0.5) if Y > X where n is a whole integer >= 1,
orX > Y (as long as the teeth fit in between the light source & sensor)
Changing the X or Y so that n switches from odd to even (or to X>Y) switches the direction the mouse thinks it's spinning. (Odd to odd, even to even or even to X>Y don't change direction.)
The worst case is Y = X * n, where n is a whole integer >= 1. The sensors can't tell which direction the wheel is turning. The closer to this, it's easier for the sensors or mouse to make a mistake.
Of course, sometimes tooth width <> gap width, so that complicates above equations.

Anyways, in your situtation, X > Y and it works, so you should be good AFA the sensors working correctly. Having enough resolution per rotation is another story.
In theory, the Windows mouse settings don't have anything to do with MAME. In reality, sometimes they seem to anyway. Generally, it's preferred to disable "Enhanced Pointer Precision", "Mouse / Cursor Accelleration", and drop the mouse sensetivity all the way to the bottom, so as to remove Windows from the equation, as much as is possible.
Also, don't forget that when you're in GAME things will be different. Yes, the mouse is expecting to see those 40 teeth per inch, and will report to Windows as such. Games will be different. When I was testing trackballs I would note that spinning the trackball in a circle would give me a *tiny* circle. Games pretty much worked okay, though still needed a bit of tweaking for optimum performance.
These two are partially related, in that the windows settings shouldn't effect mame.
Beyond that, the original game might be expecting a specific teeth/rotation ratio than what works well in windows. Mame has an "analog sensitivity" (ingame, tab, analog controls, xxx sensitivity) that can be changed on a
per game setting.

The defaults usually are for translating "standard" windows mouse resolutions to what the game expects (IOW, < 100%). Since you don't have a standard window resolution mouse with your hack, you can increase this setting to 100% or up to 255%. But as Kemmit mentioned, any setting > 100% "increases" the resolution in software: a 20 teeth @ 200% is choppier than 40 teeth @ 100%. Increasing the mame sensitivity to 100% won't hurt you, and will help a little but maybe not enough.
If you want, you can try making a higher res (more teeth/gaps) encoder wheel with transparencies. Others here have made it work, and if your search you might find some encoder wheels to download. I'm not sure if they match you wheel, though.