From memory NeoRageX was mostly hand-coded in assembly with the intent to be super fast. Somewhere between it and MAME would be something like Kawaks, but nowhere near as fast.
Without trying to sound horrible about it, if you want smooth NeoGeo emulation through MAME or Kawaks, you'll need to upgrade your hardware.
Theoretically speaking any frontend should be customisable enough to meet your needs. FEs like MAMEWah allow fairly flexible customisation of commands for an emulator. That would be the cheaper alternative, but you're going to have to do some reading of the documentation.