The contrary is true, the larger the bit diameter, the
less the bit has to work. A larger bit has more mass and is less likely to chatter or deflect during the cut. The general rule of thumb is to use the largest diameter shortest flute bit that will do the job.
For your application I highly recommend
this bit. Enter woodnet10 in the coupon code box for 10% off. That place is great and shipping is free. Whitesides are the best money can buy. You won't regret it.
If you want to save some money, I highly suggest getting a bottom mounted bearing flush trim bit.
This one, for instance, is only $17.96 shipped. I have this particular bit and it works great. The only difference is that you put your template on the bottom. If saving pennies is your thing I'd go that way. I use that bit quite often and have been very satisfied with it.
Lastly what router do you have? If it has an optional 1/2" collet available I highly recommend switching to it. 1/4" shank bits can only get to about 3/4" diameter before they start chattering badly. If that's not an option, here's a couple of bits to look at:
This is the flush trim bit. Only $12.17 shipped after the discount. A very good bit.
Here's the pattern bit (the one with the bearing on top). It's $17.42 shipped. Like I said, if you want to save some pennies I'd get the flush trim bit and simply place the template on the bottom.
One more thing to do for a nice clean cut is to use a jigsaw to trim the workpiece to as close to the original as possible. This leaves less for the bit to have to remove and results in a much smoother cut.
Hope this helps some.