Since I use interlace (768x576i) it could also be that it is the odd lines that displayed in the top half and the even lines in the bottom half.
Yep, that's exactly what it is. Unfortunately the ati drivers don't support interlace for the newer Radeon cards - I get this behaviour using both the framebuffer and SVGALIB drivers.
Annoyingly, interlace modes work perfectly in X. What I do is play everything that's standard resolution in AdvanceMAME, and interlace and vector games in XMAME (DGA mode). What would be ideal would be if one day the ati drivers were fixed to support interlace using the framebuffer.
Actually, does anyone know if this is ever going to happen? Since X supports interlace on my Radeon, surely it must be possible for AdvanceMAME using framebuffer to do the same?