Sounds like you are using a more current version of mame, since you're calling it "mameUI". In mame you can tweak each game individually, the controls, the screen orientation and just about everything else, game by game, you will need to be in mame, not your FE when you make these adjustments, so they will stick when you close mame and open it again with your FE.
I like mala, it's very easy to use and very configuarable and has tons of support, even its' own forum here! I have never mixed screen orientations in a single game list in mala, I'm not certain that you can mix screen directions in a single game list. You could create seperate game lists and have each list have its' own orientation. Good Luck!