I wish that I had the time and/or resources to develop some test cases to help nail down this bug for Northcode. My gut feeling as a software developer is that this is some sort of overflow issue. Possibly they didn't allocate a large enough array to hold the list of resolutions or something along those lines.
I'm almost sure that's the issue indeed, because after spending some time debugging HS I found that when the crash happens the EnumDisplaySettings API address is still shown in the stack, so it seems it crashes right after that function is called. The problem is, as you say, that the bug might probably be located in some external library they're using. So if a fixed array is used, it's likely to be overflowed as the number of video modes more than doubles the ones you'll expect to find in a normal system.
I uploaded somewhere in this thread a possible workaround, a modified version of VMMaker that allows to only create 32 bit modes, so this shortens the list quite a lot a probably help in some cases. I don't know if someone else than cotmm68030 actually tested it. I'll repost it within the next days in a separate thread.
Probably, the reason why different number of modes work in different systems is that the actual total number of modes changes depending on your setup, so even if we always add 120 custom modes, the driver will add some native modes, and depending on the monitor's edid some of them will be added or not.
The driver has a built in mechanism to allow the user restrict some modes, but due to the nature of the patch, in version 9.3 I needed to disable that feature, that's why we can't restrict those unneeded modes.
There is a way I'd like to explore at some point, it's to create a custom edid to override the one in the monitor, that would be an alternate way to tell the driver which modes we don't want.
At the moment, I'm concentrating in finding a way to enable custom resolutions on the fly, without rebooting. This would solve all the problems once and forever, as we wouldn't need to store a precalculated mode list any more. The probabilities of success are scarce, but anyway... I've dropped a message on the AMD board, in case there's someone on the other side:
http://forums.amd.com/devforum/messageview.cfm?catid=347&threadid=153032&enterthread=y AMD drivers have an API called ADL that allows enabling custom resolutions on the fly, indeed, since Catalyst 9.3. We have some problems however:
- Available headers (2.0 and 3.0) don't seem to work with old Catalyst 9.3
- Even if they worked, this API rejects anything below 640x480