Let me see if I understand what you're getting at...
If you mean you want to rotate/flip your desktop in Linux, and start any other program (VirtualBox, SDLMame) using those settings, then the answer is 'Yes.'
If MAME is set to not flip your screen, it will use whatever settings you currently have (i.e., whether your monitor is vertical or not, MAME will still rotate or not rotate based on its own settings). Any applications that run on X will basically see that you have a monitor with X x Y resolution, and don't care what the default orientation of your monitor is. XRandR is operating at a lower level than your applications, so they'll see whatever settings it is using.
That help?