I think you'd be better off using the mounting plate. What sticks are you using? Sanwa or Seimitsu, I assume?
My control panel is metal and I used Sanwa JLWs. I didn't wanna see mounting hardware on my sticks. I took a couple of pacman joystick mounts (from twisted quarter) and had a guy tack weld them to the underside of my control panel. I just had to trim the mounting flanges for them to fit, but it worked out great.

I also used spacers so the shafts were too high above the panel

I don't think the mounting brackets are needed but I don't see the benefit of removing them. Even if you wanna do recessed screws, why not use the brackets, otherwise your mounting holes are gonna be right on top of eachother.