Boot a game. When the title screen appears, hit 'TAB'. You should get this menu :

Use the cursor keys or the joystick to get to 'Input (this Game)', press 'enter'. You should get something like this :

Ignore the 'nach-oben', 'nach unten' stuff, that will be different for you because I'm using the german version of Mame. The important part is that you have the 'Joy 1 Up' etc stuff in the P1 up, down, left, right lines. That means that the joystick will work.
Now open the windows notepad and press the start player1 button on your CP, you should get a '1'. Press the coin player1 button on your CP, you should get a '5'. If you get anything else, the Xin-Mo encoder is not configured correctly.