You can just force whatever resolution at any time, provided it exists:
groovyume romname -resolution 320x240
However, GroovyUME will use fractional scaling still.
GBA is 160 lines tall and Groovy is configured by default to consider that resolution too low to be done as progressive. So it picks 480x480 that is an integer multiple of both height and width. If you want Groovy to allow 160 lines modes natively you need a custom crt_range:
monitor custom
crt_range0 15625.00-15750.00,49.50-65.00,2.000,4.700,8.000,0.064,0.192,1.024,0,0,160,288,448,576
(notice the remarked value, by default it is 192).