It could be a few things.
first, remove the d3d and use hwstretching to 1. This SHOULD attempt to stretch to the same resolution.
If this doesn't work. Then its probably the resolution your monitor doesn't like (MAME is trying to stretch 320x200 -> 640x400 because of exact stretching.... But your monitor can't handle it.... Or whatever)... Try forcing a resolution you KNOW will work (640x480, 800x600, ect) and try again...
or
use hwstretching. Some video cards do MUCH better with it (based on opengl instead of d3d)