I was typing this answer yesterday from my time zone, just left the office for a while and when I came back the site was on strike!
OK some more (bizarre) information.
It looks like the 15khz 320x240 is perfectly fine before running vmmaker. I was able to select it in ArcadeOSD and in gmame and it displays fine on the CGA arcade monitor. I also see over 80 resolutions in ArcadeOSD.
Once I ran vmmaker and rebooted, the 15khz 320x240 was replaced with the doublescanned one and I went from over 80 resolutions to just a small handfull, including the 4 1234x ones. Not sure how that could be messing it up.
The number of resolutions we're interested in is not the total number but the one shown inside (), which acounts for the number of *custom* video modes. I assume you checked this when booting with the arcade monitor attached, otherwise that figure would be misleading. The driver installs around 120 custom modes, so if you saw 80 total modes then it's highly possible that only 60 custom modes are being accepted.
It's normal that once you run that vmmaker the number of resolutions drops, because the spefic package you're using is configured for 'magic' resolutions. Edit its vmmaker.ini to process MAME.xml and you'll get a hundred of them, but that's not the relevant fact here.
It sounds like the active miniport driver (ati2mtag.sys) is not the hacked one.
Another weird thing, I was able to select 640x480i desktop mode in ArcadeOSD and display perfectly on the CGA monitor, but when I reboot with the CGA plugged in (and the DALRULE_DONOTPERFORMDISPLAYDETECTION registry setting that ignores monitor detection set), it freaks out and goes to safe mode with only 640x480p and 800x600p. When I reboot again with the LCD plugged in, it stays in safe mode and arcadeOSD shows the same 640x480 and 800x600 resolutions. WTF?!?! I am totally confused.
Well that's really weird, that happens when the driver fails to load and Windows picks a default one.
I wish I knew what file was causing this ridiculous behavior. What idiot at ATI thought it would be a good idea to uninstall the video driver if the monitor wasn't recognized? At this point I think I may have to reinstall windows to get rid of whatever catalyst 11 file is doing this. Calamity, is there anything in the 9.3 CRT EMU driver that might cause this? Or do I have some remnant of catalyst 11 causing it?
I believe it's not ATI's fault but some specs required from Microsoft lately, as nVidia cards seem to be doing something similar. That's why I'm hoping that at least they provided some sort of back door through undocumented registry settings. Otherwise the only way to revert that behaviour would be patching the driver and I highly doubt I have the skills for something like that.
It's highly possible that some files from Catalyst 11 are still cached by your OS. I know this SUCKS.
Something you can try is to repeat the process with CatUninstaller and after that, re-booting in safe mode, try to manually uninstall the default display driver from the device manager dialog. The idea is to eventually leave Windows in a state where it can't even recognize your video hardware, then it should happily accept our driver.
BTW I just received a new HD 6450 card, so finally I have something to test directly. I also have a Soft-15KHz dongle to test with, so I'll be able to check if it's a solution for this issue.
Well just antoher question for calamity, i guess magic resolution on arcade Monitor (hantarex) provide wide range of modeline, but , i don't have 1234X224?
I didn't include it because it was not strictly necessary, but it's just fine if you added it!