I'm not sure if it's anything at all like the Pac-Man joystick I just disassembled.
My center pole had become "fused" with the metal of the centering grommet (rust). A few shots of WD-40 and then a few swift taps on the end of the pole with a small hammer and it eventually broke loose.
Like I said though, I'm not sure if yours is similar or not, but I hope this helps.