The vertical aspect ratio option modifies the -aspect option inside the vertical.ini file.
To put this in context, back when GroovyArcade was released, most of us were used to the ArcadeVGA standard for vertical resolutions. The ArcadeVGA has several resolutions that are intended for their use with vertical games. These are all 384 pixels horizontally, eg. 384x256 and 384x288. The result of rendering vertical games with the correct 1x1 pixel scale on these resolutions is incorrect aspect-wise, resulting in a visible area that is wider than it should, but in a way it's also nicer because it covers a bigger portion of the screen.
When I coded the modeline algorithm, I made it so vertical games were shown in their correct aspect ratio. In other words, if you take a ruler and measure the visible width and height on the actual screen it should result in 3/4. However, many people complained the games looked too thin, probably just because they were used to the ArcadeVGA way.
So this option was provided, to allow forcing the vertical aspect ratio and make it more similar to the other (incorrect) way. It works by "cheating" about the physical aspect of target monitor. This makes GroovyMAME apply a correction factor that in the end achieves the (incorrect) desired aspect ratio.
E.g., if you tell GM your monitor aspect is 3:4, it will render vertical games full screen. This is the edge case.
Setting monitor aspect to 3:3 will cause games to look square on a 4:3 monitor.
A value in the middle like 5:4 achieves somewhat similiar results those of the ArcadeVGA.
etc.