Purchased the exact size hole saw I needed from Amazon since my big box stores didn't have it.
I didn't want to risk cutting that with a router for the very reason you mention.

There are two straps holding it place in sort of an x or cross fashion bent tight to the shape of the t-ball case and just a skosh shorter so that when screwed into place it pulls the tball tight against the surface.
It's just those straps, plus some minor help by how tight the lip fits in the hole, that holds it in place. Been there for 8 months (with two removals for unrelated reasons) and it hasn't twitched once during gameplay.