Ok, I now see what's going on, GM is picking 1024x768 for refresh accuracy reasons (as compared to 400x256), but AVGA's 1024x768 is a fake resolution, it does not have 768 lines indeed, unless you run Ultimarc's tri-sync utility (as I mentioned above). That's why borders are chopped.
Try using 'sync_refresh_tolerance 8.0' in mame.ini. That should avoid 400x256 from being dropped in the first place. You may need to adjust your v-size anyway, but at least the game should fit. Please post a log when done. If have a deficit of AVGA logs.