You get the most control over the process by doing it manually. Programs like GK and AutoGK use the same tools as the manual method, but they automate the process.
To summarize the manual process:
- Run the VOB's through DGIndex (formerly known as DVD2AVI) in order to generate a .D2V file. If your DVD source is film, then you'll want to IVTC (inverse telecine). If the source is NTSC video, or the format is PAL, then you don't want to IVTC. Also, DGIndex can demux the audio, which you would normally want to do.
- Write an AVISynth (.AVS) script (you must have AVISynth installed of course) to load the .D2V file into VirtualDub[Mod] via "MPEG2Source". Also, if the video needs cropping and/or resizing (which it will unless you have a specific reason for not wanting to do so), add those commands to the .AVS script. If there are any other filters you want to use for processing the video, now's the time; e.g. KernelDeint for deinterlacing DVD material originally sourced from NTSC video.
- If you're using VirtualDubMod, then you can load the .AC3 audio file that you demuxed, directly. Depending on your file size/quality goals, you can leave this untouched in it's original .AC3 format, or compress it to e.g., .MP3 (you can do this in VDub as part of the overall encoding process, or separately in something like BeSweet). As far as video compression goes, use DivX or XviD (I prefer XviD), and there are a multitude of settings you could do here. If you want a specific file size for the resulting .AVI, and picture quality is important, you'll want to do a 2-pass encode. Make sure you use the "fast recompress" setting in VDub; otherwise, if you use "full processing mode", it will undergo an unnecessary color space conversion during the encoding routine.
There is a lot to it if you are picky about the quality and/or size of the final file; far more than I mentioned in my summary. If you are not particularly fussy, use AutoGK.