You can usually stuff a few ground wires into the terminals if needed, but they might not be secured very well by the terminal screw. Most people daisy-chain grounds together (the ground wire from one button or joystick goes to another button or joystick and so on) with just one ground wire coming off the whole thing, and that wire goes to the I-PAC.
I personally use a separate terminal block to run my grounds to, and that in turn goes into the I-PAC, so I really only use one of the ground terminals to connect 5 joysticks and 23 buttons.