Just tested the Ipac with a multimeter, and the Ipac ground pin is directly connected to the USB ground wire. So, no reason not to connect all your buttons to it at once. (no current-sinking limit other than the USB port's.)
Curious, why do you say to put the joystick on its own ground loop? I would think the only thing that changing the layout of your ground loops does is change subtle electromagnetic characteristics of the wire - its capacitance and inductance with other wires. While this might be important if you were dealing with radio signals or high frequency stuff, it's not here. So all changing the shape of your ground loop does is make it either more or less obvious where a break in the wire is, if a loose connection should develop.
If we were talking about tiny gauge wire and larger amounts of current, like LEDs or lamps, then maybe gauge of wire vs. current through it would be an issue. But not for joysticks and buttons.