i have a very similar setup to you for my mame machine and cannot run 3d games at full speed/smoothly either. if you want to fix that, you would need a more modern faster cpu
ps. for some games (probably not 3d ones, i havnt tried though), you can try underclocking within mame to get them to run at full speed. a good example is 'arkanoid returns'. i get slowdown during the intro sequence for that game unless within mame i underclock the motorola maincpu to 50% (it works because mame has to emulate that cpu at 100% speed, even though 'arkanoid returns' doesnt need the full power of that motorola cpu in order to run at 100%)
some people say you may run into gameplay problems doing this but i havnt had any problems myself so far playing this game (eg. lost speed/smoothness, or sound issues)
how to do it:
set cheat 1 set in your mame.ini
load 'arkanoid returns', go into the mame config menu, then into the slider controls submenu, and set Overclock CPU: maincpu to 50%