FIX: So I'm an idiot, mb? I had to set the 4:3 aspect in mame.ini to 3:4. Duh. I'm not sure if this is because I was doing something else backwards. With a horizontal monitor, rotate_r/l just works (assuming you tilt your head 90deg

)
So, I have a cab with a vertical CRT. Windows desktop works fine at 576i, 240p, 480i (all rotated, of course). I can see the edges. Size and position are good. GroovyMAME menu is also fine.
But when I load a game, say Dimahoo for testing, the display is cut off at the top and bottom (what would be left and right on a normal horizontal monitor). The visible area doesn't overstretch the screen, i've set the size so i can see where the picture cuts off about 1 cm in from the top and bottom edges. I can also do this by adjusting the modeline in ArcadeOSD and then transferring those monitor values into an ini file. Either way nothing changes. The raster is trying to display larger than the screen size and the modeline.
EDIT: I'm actually screwing up the modeline insertion in the ini, and just adding a range not inserting a line. I'm not sure if this will make a difference given the range specs are still the same, but i'll give this a go next.
It's not a new CRT, but I've recapped the chassis to fix some geometry problems, and like i noted, i can get the 240p squares image to show all the red squares around the edge, and shrink the top and bottom in either with the pots, or adjusting the modeline.
What am I doing wrong here? Any help greatly appreciated
