Errr, i believe GroovyMAME's switchres module (that changes and adjusts resolutions for you) only works properly with crt_emudriver, and crt_emudriver won't work with an AVGA. You can still use GM, but you might need to disable switchres, which is really the point. Can you keep using the tools you already are? Ultimarc provide their own software tools for installing resolutions, don't they? I do know those tools sometimes fall over, and the usual advice is to talk to Andy at Ultimarc for support, but no one has ever posted back on here with a successful fix. It seems sometimes the Ultimarc drivers just won't install.
Sorry to be difficult, but for a complete solution you could look at grabbing a cheap Radeon HD 7450 for $30-$40 from eBay and then go the crt_emudriver route, which is a lot cleaner. And you might think about updating to Win10/11 which is going to be a lot more future-proof.