You're thinking along the right track.
Use your middle 2 poles for ground.
Use your outer four poles wired to buttons, and then to ground through the buttons.
Edit: Wait, i reread your post and now i'm confused. The way i described is so you'll have 2 mouse buttons for your track ball, and buttons 5 and 6 for p1, each pair being selectively en/disabled by the switch.
If you're looking at using p1b5 and p1b6 as your mouse left and right clicks, you'll want to wire as follows:
Middle 2 poles to p1b5 and p1b6 buttons, and then to ground
Top 2 poles to signals for mouse buttons, no switches/buttons involved
Bottom 2 poles for signals to encoder, no switches/buttons involved
This way will switch between encoder functions or mouse functions when the switch is thrown.