These settings may be good for your monitor.
In vmmaker.ini (run vmmaker and reboot)
MonitorType = "CUSTOM"
monitor_specs_0 = "15450.00-16050.00, 55-65, 3.91, 4.70, 6.85, 0.190, 0.191, 1.018, 0, 0, 288, 448"
monitor_specs_1 = "23900.00-24900.00, 55-65, 2.91, 3.00, 4.44, 0.451, 0.164, 1.048, 0, 0, 480, 768"
In mame.ini
monitor_specs_0 15450.00-16050.00, 55-65, 3.91, 4.70, 6.85, 0.190, 0.191, 1.018, 0, 0, 288, 448
monitor_specs_1 23900.00-24900.00, 55-65, 2.91, 3.00, 4.44, 0.451, 0.164, 1.048, 0, 0, 480, 768
If you want to explicitly create a resolution, edit ReslList.txt in vmmaker folder before running it, and paste a line for the resolution you want, like this:
496 x 384 @ 57.524160 system24