I have Competitions in my 3/4 panel PLUS 1/8" Lexan and they feel fine. Before you chisel out material, I would try it first in a separate test panel and see how you like the feel.
Of course, you can always add washers as spacers if you do route the material and then want to bring the handles back down shorter.