To change the orientation of the mounting plate from the long axis of the plate being on the X-axis to the Y-axis, just remove the 4 screws holding the mounting plate to the joystick base and turn it 90 degrees.
Your picture shows 3 of the 4 screws.
Scott
EDIT: Also note that there are two tabs on the pivot that keep the stick from rotating.