You're calculating wrong. That list is mostly classic 15kHz games - almost bang on 15.7 kHz, from memory. Almost all old 2D arcade games are , all the capcom ones, neogeo, all the taito ones. Sega system one is like 512x240. There's a few outliers in mame like Blazing Lazers which is 1288 x 256 or something, but even that isn't too far out.
Using height x 1.05 can approximate badly sometimes. You would normally be using Vtotal to calculate, as the sync circuitry doesn't care what the guns are doing, it's still drawing lines. So for a Neo Geo game:
H-freq = Vtotal x refresh
15.626 kHz = 264 x 59.19
It's actually really unlikely that a best-of mame list, even your own best-of, will need anything more than the standard user mode + super.ini file contains (MK games aside, IIRC). Rather than trying to push for range limits, grab your games list and load them one by one in GM, taking note of the infoscreen that splashes up telling you the game's native resolution, and then the resolution GM has picked. If this looks right there's not a lot for you to adjust beyond tweaking the horizontal size or position if those are slightly off.
The range limits are set in GroovyMAME as it's inbuilt switchres utility calculates a modeline on the fly. It grabs a mode with the right number of vertical lines and re-writes it with the exact refresh rate of the original game, subject to the constraints of the range line. This way you get the most perfect reproduction possible. There can be more detail than this, but using super resolutions is the easiest/best way to do it. You install the super resolutions, and then in mame.ini set resolution to 2560x0 and super_width to 2560. This tells GM to do the above. If you're confused by super resolutions, have a look at section d) of that guide i linked to above.
By default, GM is set not to stretch the image in the vertical direction when it finds and calculates a modeline. This is critical for perfect visual reproduction. A game like SF2 on the CPS1 system, at 384x224, will pick 2560x240 if that is the mode with the lowest number of vertical lines you have, and simply leave 8 blank lines each at the top and bottom.
I would guess that your problem is the TV being sensitive to the output of the VGA-to-Component converter, not the video signal itself. I would not presume other equipment would fix this, either.
If you do want to play around further, be aware that your TV probably won't go far below ~15kHz if it goes that far, so you'll want to test that. Also, it sounds unusual that you'd be finding games running less than 15kHz. Even ancient Dos games running CGA were 200 lines at 70Hz refresh - so still 15kHz. They didn't really make monitors that ran less than 15kHz; at least, not anything in the 80's. It's only us emulator users that really try to screw with modelines that want frequencies that low, and even then those are edge cases like the 384i mode i mentioned - which looks like balls and is really squashed down anyway and totally not worth the effort.
EDIT: yeah ok i just watched the video. That is pretty bad. Your PVM-20M2 will take component also - does it work fine with the transcoder, or show the same issue?