If you go into the text screens (the blue and red one), the input acts funny, too. (tested with 0.128u3 with mouse.) The crosshairs not only jumps from one side to the other, but part of it is slower than the other, for both X and Y axes. I could keep the crosshairs as centered as possible with x adjustment of -35, and y adjustment of -2F, but...
The game still had pretty much the same symptoms.

Hard to tell how much is valid with only a mouse, though; there's no center reference (testing at work isn't as complete as at home). Do you mind installing a more current mame in a different folder, and testing with your joystick? I'll see if I have time tonight, but.... :shrug
As to the digital feel, there are two factors.
First is a "speed of sight" dipswitch; I think this sets the fastest speed the sight can move. If the control moves faster than this, the game limits the speed, and stops when the slowed sight "catches up" with the joystick's position. (It acted that way with a mouse.) Gives a
very "digital" 8-way feel, but still is really analog.
Second is mame's default mapping. By default, joysticks are mapped to both the analog part and the digital2analog part of analog inputs; not too bad default since windows doesn't tell apps if the joystick is analog or digital. However, you should unmap what it is not: for analog joysticks, unmap from the inc & dec inputs leaving only the analog input, while for digital joysticks unmap from the analog input leaving only the d2a inputs (inc/dec). Unmapping digital joystick isn't required, but unmapping analog sticks from the dec/inc is needed for true analog control with analog joysticks in mame. (Hint, use "Input (general) or ctrlr file so you only have to unmap once for all games.)
FWIW, there's a
bug at mametesters that looks on this problem, but a min/max is currently in there now, so the guessed problem doesn't seem to be the problem.