What monitor type do you have
I tried in two diferent machines with diferent monitors. One is QuadCore Q2800 with a tft Samsung syncmaster 226BW (22') and the other P4 3GH with a 15' tft with no name. The samsung is running in a NVidia Gforce GTS 250 and the other in a Nvidia Gforce 6600. Both with Debian amd64, the 15' is a clean install.
what does your entire xorg.conf
Attached a copy
linux version/distribution
Debian amd64 with kernel 2.6.32-5-amd64
possibly logs of /var/log/Xorg.0.log with your "Device" section containing this Option "ModeDebug" "true" which will show all the modelines being used.
Attached a copy
It sounds to me like you might need better modelines
Who can I do this?
but also it could be that artwork is enabled or available (even when it's disabled if it is seeable by mame it'll offset things). Also try either -video opengl vs. -video soft, one or the other might act different with this offset.
It is not a mame problem, I have the same with mednafen, stella, etc.
Also output of /var/log/dmesg log too, might be interesting
Nothing here
are you using a newer kernel with DRM and KMS switching?
No idea about this.
Ah ok, if it's TFT monitors then modelines shouldn't matter, in fact probably xorg.conf doesn't matter too much but mainly it's probably how the screen is being rotated. Instead of doing it in xorg.conf you might want to try xrandr to do it, also you might want to try and use the newest Xorg version like in Ubuntu 10.10 which is way newer than the one your using (1.7.7) Ubuntu uses 1.9.0. You might not even need an xorg.conf because newer versions of Xorg do better without one, then when your inside X Windows run this command (can put this into the window manager startup like .xinit possibly, or however your X windows setup is)...
xrandr -o left
This will rotate it left, or right if you need it right, and normal to get it back to normal.
It might do it better, I'm at least guessing that is the issue, or possibly that and a newer version of X Windows combined would fix it. Since it's in all emulators and your using a normal TFT monitor I'd guess it's all to do with the Xorg setup and version, one or the other or both.
Update:
Also, since your using the Nvidia proprietary driver, you have to replace the CCW option with this...
Option "RandRRotation" "on"
Which allows xrandr to work
And also since it's the Nvidia driver, which I'm all too familiar with because I run it on my desktop, you will need the config. One thing though, is if you use Ubuntu 10.10 the newer Xorg has the Nouveau drivers which are worth a shot if you can't get it working right with the above suggestions. The Nouveau drivers are more advanced in a lot of ways and actual native X drivers, not proprietary and a bit more in sync with newer Xorg methods (like no config file necessary). The one disadvantage is if your using OpenGL heavy duty graphical stuff or playing video back on the system, not mame stuff, but other things then the Nvidia drivers beat the Nouveau drivers in acceleration like playing video back in 1080p without tearing (which is why I still use the Nvidia one instead of Nouveau, I tried but it just couldn't do what I needed for a desktop, yet).