The Startech device page you linked contains some important information in 2 of the the comments, though they slightly contradict each other:
- "VGA2VID accepts VGA input up to 1600x1200 resolution and outputs S-Video or Composite at 640x480."
- "The VGA2VID supports PAL: 576i and NTSC: 480i. An interlaced signal draws half a frame (called a field) at a time. NTSC 480i supports a frame rate of 30 fps. The field rate is 60 Hz. The PAL standard of 576i supports a 50 Hz field rate, and the frame rate is 25fps."
This means the device outputs video in a fixed mode (either NTSC or PAL). Everything will be squeezed/stretched to fit this mode. For best results, you should choose a desktop mode to suit. If you are using an NTSC TV (USA), then choose a 480i video mode for desktop. Despite the top comment above, you should try 720x480i as this is actually the NTSC standard, not 640x480. Have a play and see what looks right on your TV.
You should also consider restricting MAME to output for this video mode only. However, you probably won't even have a 720x480 or 640x480 mode available without special drivers/software. Unfortunately, CRT_emudriver won't work with nvidia cards like yours. Some people use CRU instead, but it doesn't have all the cool features as Groovymame/crt_emudriver. Luckily compatible video cards are cheap, so I'll leave that to you. Check the Groovymame forum branch for more info.