Ok, i'm going to post some screen shots of how i'm doing this, because maybe I am doing something wrong. Hopefully Calamity can chime in on the Arcade OSD thing, because i think that is where all the confusion is coming from.
Modifications: I create a user-mode-640-only.ini file, with the only option:
640 x 480 @ 60.000000 desktop
Step 1: I'm launching VMMaker, and selecting NTSC as the option on monitor tab. (Using default values in monitor.ini that is pulled in, which is set to:
monitor "ntsc", "NTSC TV - 60 Hz/525", "4:3"
crt_range0 15734.26-15734.26, 59.94-59.94, 1.500, 4.700, 4.700, 0.191, 0.191, 0.953, 0, 0, 192, 240, 448, 480
Step 2: On the video card tab, I select my video card from the drop down, select the analog output and enable EDID emulation.
Step 3: On the user modes tab, I browse to my modified user-mode-640-only.ini file that has the single user mode.
Step 4: On the mame tab, i browse to GM executable. Leaving XML processing off.
Step 5: I hit OK, and it exports the settings to mame.ini
Step 6: I hit Generate Modes, and it generates a single modeline.
Step 7: I hit Install Modes, and it installs them into the driver.
Step 8: I launch Arcade OSD, and I see two modes (im guessing one is from EDID emulation? I'm not sure why im getting one at 30 and one at 60)
Step 9: I go into 640x480@60 mode, and then copy the modeline to clip board. I get this output:
modeline "640x480_60 15.73KHz 59.94Hz" 12.21 640 664 720 776 480 487 493 525 interlace -hsync -vsync
crt_range 15724.54-15744.54, 50.00-60.00, 1.966, 4.586, 4.586, 0.445, 0.381, 2.034, 0, 0, 240, 135, 480, 270
Why does crt_range this have 50-60 VfreqMin-VfreqMax? This is not the modeline i'm using, but this is what Arcade OSD outputs. I think this is where all the confusion came from, as this obviously looks wrong. Even the HfreqMin-HfreqMax is slightly off as well.
In this state, I cannot get a signal on my Emerson CRT tv, and I get an unstable signal on the Trinitron CRT (which is what I originally posted this thread for)
On both TVs, a PS2 console with S-Video output gets a clear signal without any issues.