Well, the mode I'm using has a dot clock a bit over 8Mhz, and the same freeze problems occur using a squished (big front/back porch) 240p mode with ~13Mhz dot clock, so I don't think that's the cause. Good to know that the issues I had going below 8 were due to the Windows drivers, and the MAME resolution workaround will be handy if I ever get the rest to work.
I haven't been able to get it to display anything but 640x480 interlaced without using Powerstrip, even though all the 15khz modes are listed when looking with arcade_osd and other tools. Lower resolutions it will put centered on the 640x480 output with huge black borders, and higher resolutions it goes into panning mode around the larger desktop. This includes booting up with only the component connector connected. I think at this point it's something to do with the driver trying to treat it more like an s-video tv out than vga, and it may not be something fixable.
Dxdiag is reporting acceleration enabled for d3d and ddraw. It completes its test fine as long as I haven't switched it into a 240p mode with powerstrip, in which case it will complete the first two tests that are overlaid on the desktop then freeze the system when trying to change to a full screen resolution like everything else does.
I'd be just as happy sidestepping all these Windows issues by going with Linux, but I haven't the slightest idea how I'd enable the component video out on the card under Linux, if it's even possible. It seems like the Windows drivers is somehow triggering the switch to YPbPr video out of that port, so unless the Linux driver developers have figured that out there may not be a way. The GroovyArcade live disc just looks like it's outputting RGB over the component connector, for what it's worth.