If you do a few test holes, to find out the exact bit sizes you need to get the right size finished holes, that will work great.
I personally prefer forstner bits to do it because I like all the curly-cue shavings they make, vs. the dust from the router.
I've also cheated on a bunch of solid wood panels, and just drilled a 1-1/8 hole, and jammed the buttons into them.
Buttons don't get the upward stress that you would with a joystick, so they don't pop out of the holes.
Yet another option, especially if you are using the long buttons, is to make the CP a little thicker by creating a laminate.
I don't know the exact thicknesses you'd need, but you'd need to drill pilot holes with both pieces attached together, then drill the large hole completely through the top piece, and the small hole through the thinner bottom piece.
Glue them up, and you should be able to screw the switches directly to the bottom of the bottom piece.
I build alot of my CP's as laminates because that allows me to create more complex designs than you can with simple routing.
An example of this is the recessed dust washers.
If you just rout them, you have to cover them with plexi.
If the top layer of your CP is 1/4" plywood, you can cut the hole necessary for the joystick itself into the top layer, and then use a hole-saw to cut the second layer.
This also allows you to put T-nuts into the second layer, and cover them with the top layer.
Oscar has a really good tutorial on it
HERE.
I usually use wood where he used the plexi on the top, and just glue it to the layer below.
That lets you use putty on the seam, and just sand the whole thing flush.
What it doesn't allow is for taking it back apart, but that's never been an issue on the CPs I've built.