Some interesting updates, definitely getting interesting, I've been talking with the ATI Radeon driver guy for the Linux open source stuff.
First I've figured out through his help that the xorg modeline stuff does work but you have to avoid using any Modes configuration in the Screen section of the config. Those will write hardwired modes you will not be able to override with your custom modelines, and in fact work against you by making duplicate modes but with 60Hz refresh rates. Basically the xserver does add our custom modes but when we add them in the screens section the driver takes those names and gets HxW and creates a CVT modeline using 60Hz.
So you don't have to recompile the ATI driver, besides the lower dotclock limit of course to make it allow less than 12Mhz. Yet the PLL clock setup of course involved in that is needing to be tuned for lower resolutions with arcade monitors, same in the DRM layer with Kernel Mode Switching in newer kernels/ATI drivers if enabled. More on that in a bit.
Second the xserver may have a bug, where the default modes it creates, some of which don't conform to our Vert/Horz freq range values. These can't be overridden by xrandr at the moment because they are unwritable, and it seems like a bug. So that is the main reason the xserver needs my patch right now to remove the adding of those default modelines.
Third I'm working on a script that basically will get all the stuff needed to build an Xorg server/driver system with OpenGL and DRI/DRM all correctly setup and will patch the sources with changes we have to make for arcade monitors (at least until they are actually applied to the X stuff officially). My goal is to try and get the actual Xorg/DRM/Linux kernel all adapted to arcade monitors so it works in all distributions by default.
That is the interesting part, I installed the newest Ubuntu 10.10 to test since it uses all the newest Xorg and DRM stuff. It actually seems to do the Vsync stuff correctly, which is a plus. The minus is that it has issues with of course the PLL Clock values and some oddity with mame not being able to switch to the new modes (I can use xrandr to do it) and so it isn't centered. That part is probably minor through configuration or maybe some patching to allow it to work right. It was all unpatched just using the distribution packages, including the mame version, I'm going to try and make this custom build thing be able to work there and find out more that way so it would be easy to have others do to through the script (not have to distribute big distros since just a script and people run it to build the distro themselves, which really is just a separate tree for the Xorg stuff and anything else necessary).
So there's a flicker every so often of the monitor in Ubuntu even with a good arcade modeline that works fine in my normal Gentoo setup. Ubuntu is using the KMS code in the Linux kernel instead of UMS, which I'm guessing is the difference. Alex Deucher (Radeon driver guy) says that this very likely is the PLL clock dividers and to try the older ones, see if that fixes it. So this brings back an interesting thing about how we've always hacked these drivers to use a lower min_clock value and that is all we have done. We have never tuned the PLL clock dividers to actually be optimized for arcade monitors. This sounds like something that could make Linux be the only way to actually output the best possible signal to arcade monitors and something we could never do under Windows (unless we hacked the driver/firmware bios I guess).
Another interesting thing about Xorg setup, in your monitor section you really need to make the identifier the name of your card output, like DVI-0 or VGA-0 etc. It doesn't know how to really link the monitor section to the card output very well, usually just picks the first one per monitor section as it reads them. This may be another reason why modelines often don't apply properly, you have to setup the xorg.conf file just right for your monitor/card names and ordering.
Here's my xorg.conf which can load the custom modelines without my patches, and seems like the proper way to set it up...
Section "Monitor"
Identifier "DVI-0"
VendorName "Wells Gardner"
ModelName "D9800"
Option "DPMS" "false"
Option "MinClock" "1Mhz"
Option "MaxClock" "90Mhz"
HorizSync 15.25-38.00
VertRefresh 47.00-75.00
Option "Primary" "True"
Option "PreferredMode" "640x480x60.10"
UseModes "ArcadeModes"
EndSection
Section "Monitor"
Identifier "VGA-0"
VendorName "Lilo"
ModelName "Lilo"
HorizSync 15.25-38.00
VertRefresh 47.00-75.00
Option "RightOf" "DVI-0"
Option "Primary" "False"
Option "PreferredMode" "640x480x60.10"
UseModes "ArcadeModes"
EndSection
Section "Device"
Identifier "Card0"
Driver "radeon"
#Driver "nv"
#Driver "nouveau"
VendorName "ATI Technologies Inc"
BoardName "Unknown Board"
BusID "PCI:1:0:0"
#Option "ArcadeMonitor" "d9800"
Option "EXAVSync" "yes"
Option "EnablePageFlip" "on"
Option "ModeDebug" "true"
Option "NoDDC"
Option "IgnoreEDID" "on"
Option "ForceMinDotClock" "3Mhz"
#assigns the output DVI-I-0 to Monitor0
Option "monitor-DVI-0" "DVI-0"
#assigns the output DVI-I-1 to Monitor1
Option "monitor-VGA-0" "VGA-0"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "DVI-0"
DefaultDepth 24
SubSection "Display"
Viewport 0 0
Depth 8
EndSubSection
SubSection "Display"
Viewport 0 0
Depth 16
EndSubSection
SubSection "Display"
Viewport 0 0
Depth 24
Virtual 1280 480
EndSubSection
EndSection
Section "Modes"
Identifier "ArcadeModes"
Modeline "640x480x60.10" 25.240385 640 656 752 800 480 490 494 525 -HSync -VSync
EndSection
Also I've gotten switchres to build the modelines file and will release a version with that capability so in Windows you can have it write out a Soft15Khz custom modelines file as you play games, and apply/reboot and you'll slowly build more modelines that match the games you want to be more exact (probably need to trim/edit of course if limited to certain numbers of modelines).
This X Windows build is looking interesting, I'm still getting the base Xorg and all friends with DRM/Mesa to compile then I'll look into possibly SDL/Mame and perhaps a Linux kernel. Maybe it'll just be a simple small distro that really is just the bare bones X server and stuff needed to run mame, dynamically downloaded patched and built.