JAMMA uses a "common ground" approach which is incompatible with the matrix arrangement the encoder you have seems to use. If you could post a wiring diagram for how your encoder is supposed to be wired (should be provided by the manufacturer), that would help.
More info:
On JAMMA, the first and last 2 rows of contacts are "cabinet ground". Where exactly they all run is somewhat ill defined. On many cabinets, the ones at the lower numbered positions (near the other power wires) run back to the power supply as power common, and the high numbered positions (near the joystick and button wires) are used as the common lines for the switch inputs on the buttons and joysticks, but some cabinets wire it other ways. Midway, for example, tended to put 2 wires from the power supply on positions 1/A and all the way down on 28/FF, using 27/EE for control common and 2/BB to daisy chain power common to other things in the cabinet.
Since the function of each individual "ground" position is not defined, it's also somewhat common for the cabinet side to connect them all together. Most arcade PCBs do this on the PCB, anyway. Since the "ground" line for each individual switch is not available, you can't use JAMMA with a matrix configuration that requires a defined pair of wires to be used for each switch. It can only be used with a true dedicated input/common ground encoder.
The "ground" wire running to the monitor is "video ground". To help keep out interference patterns, there is a dedicated pin for this on JAMMA (14). It should not be connected to any other "ground" via external wiring. The arcade PCB will connect it properly for you, or you should connect it to your video card's "ground" lines if you're using a PC with a breakout cable.