Finally after some drama: some success!
After ruining my plexi twice I decided to go for polystyrene this time (they didn't have thick enough lexan at the local hardware store). This tends to melt a bit faster so I had to drill a bit slower, but doesn't crack as easy as plexi does.
Here are some photo's of drilling the first plexi panel (used somewhat the same proces for the final one)
First I spray glued my cp template onto the wood and pre drilled the holes with a 3mm drill:
Next I placed the plexi panel on top and drilled the 2 holes for the joysticks so I could mount the plexi to the wood with 2 hex bolts.
After this I drilled all holes into the plexi as well using a battery drill and drilling from the back (through the existing holes in the wood first)
Next step: drilling the buttons. the happ buttons went just fine, but with the smaller admin buttons I messed things up.
So as mentioned I bought a polystyrene panel and redrilled all the holes using the wooden cp as template:
Next step: routing the edges with a flush trim bit and rounding them after that.