Well, I did some more reading through the AdvanceMenu code and docs, and now I understand why none of what we tried was working.
The program wants all of the arguments that you're passing to MAME to be specified as one parameter on the emulator line in the config file. If you specify more than one parameter after the executable on the config line, you get an Invalid Argument error.
If you want to pass more than one switch or option to MAME, you have to put the config line parameter in quotes, because you'll have a space between the options, and without the quotes, AdvanceMenu sees it as two parameters. (The examples have quotes even if they only pass one option, but they don't appear to be necessary in that case.)
Since the program splits out the single parameter by spaces (or tabs) into "tokens" (arguments to be passed to MAME), and then does substitution on the %o[] tokens, no %o token could have a space in it without confusing either of the previous steps.
Fortunately, any number of tokens within the config line parameter can be %o tokens. That means that although you can't tell a single %o to create "-or vertical", you could either put the -or before the %o, or specify the -or as one %o and the desired orientation as another, e.g. %o[-or,-or,,] %o[horizontal,vertical,,]
This still has the limitation of not allowing quoted parameters like -view "Cocktail Artwork", but it's mostly workable.
So, with all that in mind, I wanted to see if I could find a solution for sometimes wanting to rotate the monitor and sometimes not. This is what I came up with:
You create two emulator entries pointing to the same executable in advmenu.rc. The first follows whatever orientation AdvanceMenu is in, and the second lets the game determine the orientation. You can flip between the two emulators/lists using F6, or whatever key you assign that function to.
Here are the entries on my test setup:
emulator "GMAME Follow Me" mame H:\games\gmame_0178_x64_a2\mame64.exe "-or %o[horizontal,vertical,horizontal,vertical] %o[,,-flipx,-flipx] %o[,,-flipy,-flipy]"
emulator "GMAME The Game is Boss" mame H:\games\gmame_0178_x64_a2\mame64.exe "-or rotate_l"
(rotate_l might need to be rotate_r on your setup)
I hope that gives you the flexibility you were looking for. One downside I noticed is that the program has to scan the game list twice, so it takes longer for the menu to appear when you start up. Also, you might see each game in the list twice until the first time you press F6 (because both emulator entries would be enabled by default.)