Ok, forgive me for going over some stuff you may already be all over, but...
Do you have the right modelines installed? Run ArcadeOSD, and see what's in the list of video modes. Is 384p there? If not, you can't use it, and who knows what's going on when you try. You have to install all the modelines you want with VMM first. You do this by editing the "User Modes.ini" file, and the monitor.ini file in the VMM directory. The latter file is where the monitor ranges like NTSC and Arcade15 are stored, but read the top of the first post in this thread here and see about possible ranges below (the text in purple up the top tells you what the numbers mean) -
http://forum.arcadecontrols.com/index.php/topic,116023.0.htmlAfter you install the modelines, you can tweak of modify them with ArcadeOSD if you want/need to.
RE: the BVM, and don't take my word for this because i've no doubt forgotten something if I'm not just wrong, but what ranges does the BVM-D24 sync to? It does 15kHz (240p/480i), 31kHz (480p), and 45kHz (720p), and also 33kHz for 1080i, but i don't think those bands are continuous. As in, for 15kHz (really 15.75kHz) it'll probably sync to anything between 15.00kHz to about 16.30kHz, but NOT to anything like 14kHz or 17kHz.
The 384p(rogressive) mode is 25kHz. This may fall outside what the monitor is capable of, hence things going wrong. (25kHz relates to no video mode ever used in broadcast TV, and that's what the BVM's were designed around.) It's also possible that the sync bands are quite broad, but that near the edges of each you start getting vertical size issues, the picture being too tall or too short for the screen. Note that i'm not sure this is what's going on here, and you should check it out for yourself. There's nothing stopping you setting whatever monitor ranges you want in the files above, generating the modes with VMM and then installing them. But that doesn't mean the monitor can handle what you told VMM it can. It won't hurt to try, briefly, but remember to remove the mode, or reinstall a safe list later - trying to drive your monitor outside its range again and again is not healthy. There are ways to trick or get around this, like extending the v-blanking and using a 384p mode that'll fall close to 31kHz, but then you'll need to deal with sizing issues i'd say. First of all, try to use the information in the first paragraph above when it comes to manipulating and installing modelines. When you're happier there, come back to extending 384p modes.
Lastly, Model 3 is mostly/all 3D games. It's not going to hurt much to run those at 480p, or probably even 720p. 2D games/pixel games is where 15kHz is required.
And btw you really want to be running GroovyMAME in 240p for most games, probably using a super resolution. 480i is nowhere near as nice, if that's what you're doing now. Go back to the setup guide for your card on Calamity's site -
http://geedorah.com/eiusdemmodi/forum/viewforum.php?id=2(EDIT: Not sure why i kept typing 396p, should all read 384p)