Are you using wood?...
spade-bit? joystick hole should be the same as a button...
jig-saw? (trackball)
Check with Harbor Freight...they have tools cheap...Maybe also check around pawn shops...

using a chisel is risky...
1. You could end up gouging too much...and chance of being uneven...
Dremel kit...then buy the router bits...
Know any friends you can borrow tools from?