On my minipac, the outside pins were power and ground. Pin 1 was power, pin 4 was ground. From there, you just need to 'play' with the X1 and X2 (and Y1 and Y2) leads until you get the right movement.
For the record, your colors do match my colors. I'm not using the spinner header, as I do not have a spinner. But both of my trackball leads had power and ground on the outsides.
No, it won't matter if you hook them up incorrectly. Just to be safe, though, don't have anything else hooked up to your minipac, and don't test your setup on an expensive machine -- use an old machine for your testing. That way you won't ruin anything on the off chance that something does go up in smoke.
Hope this helps!