the problem is he's turning off MAME's automatic rotation, which is the feature that causes rotation to orientate to the desktop and instead saying "I want to do it manually"
Some games with vertical monitors had them rotated 90 degrees, some had them rotated 270 degrees. One of things that contributed towards Billy Mitchell's videos being conclusively worked out as emulation based is that older versions of MAME actually got that wrong, treating it as a game with 90 degree rotation rather than a 270 degree rotation, thus meaning the video scan traveled in the opposite direction and different parts of the image drew at different times.
Once you turn the option to adjust to screen off, you're on your own, either you have to manually specify the rotation you want for each game, or you have to make it possible for your monitor to rotate in either direction.
MAME is doing what you're asking it to, but since you have an exotic config, where your physical monitor is rotating (MAME has no way to know this - your monitor does not report back that it has been rotated) you end up on your own.
FWIW A small handful of games (usually gun ones) even used a mirrored display, with the natural rendering being y flipped even on a horizontal screen.