right, there's absolutely zero speed difference if you remove drivers.
the driver did slow down in the last year or two due to fixes that were needed for the emulation of the more complex titles like Final Fight Revenge and Radiant Silvergun. There were also fixes to things like the mesh rendering for Die Hard Arcade which contributed to the slowdown too.