Got it working, it is "quadrature encoding" after all, which is great, wooohoooo!
Only now I understand this sentence:
- "Quadrature output? More like the encoder
on a mouse axis than the serial output."
That is exactly it, thank you Ed_McCarron! It turns out every mouse is actually a combination of 'arcade encoder circuit' (quadrature encoding) and 'serial protocol encoder IC'. And so, $0.10 old mice contains everything you would ever need to build your own rotary joysticks, whether to work with actual game PCB or MAME.
SUMMARY:
a.) arcade trackballs and spinners circuits are simple 'quadrature encoders'
b.) the simplest PC mouse is practically: 2x 'quadrature encoder' + "OptiPac"
In other words, this "quadrature encoder" is there (in spinners, trackballs, mice...) not so much as a "choice" as it is underlying mechanical/digital principle of all these types of encoders. It is there to distinguish between clockwise and anticlockwise direction, but if the direction was not important it will all come down to simply counting impulses - "light goes on, light goes off..."
In my test I could use optical just as well, but this mous also had a mechanical encoder wheel for the scroll-wheel button and that required a bit less soldering to wire up. Basically, you could even make it spin if you took +5V through two wires and "drummed" over pins Loop1 & Loop2 (A & B) in this rhythm:

What constitutes quadrature encoder?

It can be optical and mechanical, but not not like "mechanical rotary joystick". If mechanical, then it is a single part in itself, that looks very much like potentiometer, has 3 legs: on one leg it gets +5v and on the other two (say A & B) it passes this +5v as the wheel turns and brushes touch "off-phase" contact-teeth inside, where rhythm "A-B-A" would mean clockwise, and "B-A-B" anticlockwise, or something, but in any case that is all there is to it, the game PCB does "de-coding", i.e. counting, and all the rest.

http://www.trimoor.com/rotary_joystick/That's not mine btw, I found that web-page after everything else, but in any case it is similar to how my mechanical encoder looks like, and exactly what I am going to do with my joysticks now. I will also keep the rest of the mice to make PC/MAME adapter (OptiPac) for this new "ARCADE" spinners of mine.
...and, the optical 'quadrature encoders' are made of 3 components:
- slotted wheel
- emitter diode (2 legs)
- 2x receiver diodes (3 legs)

Yes, there are actually two receivers behind each wheel, that's why 3 legs on that one electrical component and what "quadrature encoding" is all about. Depending on the direction of rotation one sensor will get light before the other, and so we again have the same result as with mechanical encoder from above, where +5v comes to pins A and/or B either as "A-B-A" or "B-A-B" rhythm. -- The rest of the mouse is the same thing as "OptiPac", it's encoder IC that counts and converts these +5V impulses into PC serial protocol packets.
EDIT... heh, it's all explained here in greater detail:
- Everything you always wanted to know about optical arcade controls,
but were afraid to ask: http://www.zumbrovalley.net/readpost.php?artid=1This was incredibly simple information, which makes it all more surprising why is it not more obvious/documented/underlined.