To get the exact resolution without stretching for a game, this is the basic formula (which is really the same as the vertical game on a horizontal monitor formula, but you technically using the games width in one and height in the other). You basically want the resolution height to be as the original horizontal games width, and the height to be the correct proportion to that (3:4, since your reversing the monitor aspect)
resolution_width = game_width * (4.0/3.0) / (3.0/4.0)
Example here for mario...
# Horizontal monitor resolution:
# mario 400x256@56.88 15.7000Khz
ModeLine "400x256x56.88" 8.164000 400 416 456 520 256 257 260 276 -HSync -VSync
# Vertical monitor resolution
# mario 400x256@59.19 16.5128Khz
ModeLine "400x256x59.19" 8.586647 400 424 464 520 256 259 262 279 -HSync -VSync
Without any stretching or aspect ratio control, the second resolution will display it nicely on a vertical oriented monitor.
In Mame you may need to change the rotate settings too for getting it the right direction, basically -rotate and -rol seem good to add to mame.
This is what I've come up with in my switchres program and it seems to work good in testing here on my monitor, with the help of Calamity he's come up with the basic formula which is great for both vertical and horizontal games. You can change the 3:4 (which is always the opposite of your desired monitor aspect ratio, like 4:3) to things like 9:16 for an HDTV wide screen, or 3:3 if you want a more square picture, or 5:7 for a slightly taller game (or wider if it's a horizontal monitor with a vertical game). It's just thinking opposite of your normal vertical on horizontal monitor calculations.
Hopefully that information helps someone, was interesting when getting this feature working in switchres to see how the calculation actually worked to avoid any stretching.