Three ifs:
If you are changing those settings but nothing happening, then maybe you are editing the wrong mame.ini
If all else fails: start GM into the default GUI. you can navigate to "Configure Options/Advanced Options" and change rotation there. Don't forget to "Save Configuration" before exiting out.
If curious, you can then check your mame.ini "CORE ROTATION OPTIONS" and see what was changed.