You have to use the FIRST Output on ATI cards. Most likely it is NOT the VGA port, but the DVI port on modern cards.
Thats why you have to use \\.\DISPLAY2 on your setup.
Give it a try with a DVI-VGA adapter.
That would make sense if I couldn't see any of the custom resolutions under the \\.\DISPLAY1 list - but I can see around 15-20, just not all of them.
...I've just spent another several hours working on the problem. I removed the existing Soft-15kHz settings, downloaded Catalyst 9.6 (which came out this morning), removed Catalyst 9.5 and deleted the registry entries for the card.
I then installed the new Catalyst 9.6, backed up the registry key(s), installed my custom user modes using Soft-15kHz into both display entries (even though I am only using one attached monitor) and rebooted.
The good news - no black screen this time, so I guess they've fixed that in 9.6. The bad news - MameUI behaves exactly as before. \\.\DISPLAY1 only shows around 20 of the custom modes and \\.\DISPLAY2 shows all 40 (or so) of them. (This must be a MameUI bug).
320x224@59Hz mode has borders top and bottom still and appears to be identical to a 320x224 game running in 320x240 mode. (Remember that I made this mode with the NVidia card attached and it worked perfectly - with no borders).
I shut down the computer, swapped the monitor cable from the DVI port with dongle, back to the dedicated VGA out port. Restarted the computer and results were identical.
Altering the mode of \\.\DISPLAY1 or \\.\DISPLAY2 changes the mode that the game(s) run in, no matter which output port the VGA cable is connected to. This is probably a behaviour of the ATI driver running in single display mode.
I don't think the ADVV tool will work with ATI 4000 series cards, so I'm unable to experiment with the values of the 320x224 mode - but if the mode displayed correctly with the NVidia board, it should work with the ATI, right?
I did run AdvanceMame and the 320x224 mode it generated filled the screen without performing any stretching of the image. Too bad it's no longer kept up to date.
I think the only thing left to do is see if there is a verbose debug output in MameUI, to double check that it is at least attempting to put the screen into 320x224 mode. If it is, then the problem has to be ATI's fault - they might be messing around with the mode values.