I think all the newer nvidia cards support rotation, provided you install the full blown tools rather than just the bare bones driver. I have an nvidia card (and it's not an expensive card either) for my vertical monitor and it rotates windows for me. I love it!
Back to your original question though, yes, you can run mameui in the same way as you'd run mame from the command line. I run mameui (from the mala FE) because in the rare case where I want to change game settings, I prefer to do it from mameui. Command line parameters are the same as they are for mame.