Suggestion:
Dont use the GFX hardware utility to switch things. MAME and MALA have their own internal things for this. Since you are making windows fit into a rotated view you may be confusing MAME. Make the ATI setting go back to horizontal, and use MAME and MALA settings to make things be vertical...
Basically, start with the thing you need (MAME) then work your way out... Just try it with everything else being "horizontal/origonal" but with MAME set the other way.
EDIT: Sorry, MAMEwah, not MALA