The only range of yours that allows for SwitchRes to pick a 288 lines res is this one:
"SwitchRes: Monitor range 15250.00-16500.00,40.00-80.00,2.187,4.688,6.719,0.190,0.191,1.018,0,0,224,288,448,576"
Hence the 53Hz, due to the 16.5 kHz limit.
Now in ArcadeOSD you get 288 lines too, the difference being an horizontal freq of 24 kHz. This freq allows 60 Hz refresh but also ads too much (non visible) lines, see below. That's why the visible area is smaller.
"DALDTMCRTBCD2560x288x0x60 - Modeline "2560x288_60 23.94KHz 60.00Hz" 81.59 2560 2800 3048 3408 288 335 339 399 -hsync -vsync"
On the contrary the 16.5 kHz range does not allow for enough lines so the screen does not fit your monitor.
"DALDTMCRTBCD2560x288x0x60 - Modeline "2560x288_60 16.50KHz 53.05Hz" 54.52 2560 2680 2936 3304 288 291 294 311 -hsync -vsync"
The reason why the 24 kHz res exist in ArcadeOSD but is out of range in GM might lie in a mismatch between VMMaker and GM config files. Post your VMMaker.ini if you will so we can check.