It most certainly will allow non-VESA standard modes. I've got a Matrox G100 that I run at 1440x480 interlaced (NTSC compatible) all the time. That is decidedly NOT a VESA standard mode, nor do my timings conform to VESA GTF.
For a standard res monitor, 49-61Hz is usually a valid vertical sync range, and 14.8-15.75kHz is usually a valid horizontal sync range. If you specify those, then X will only attempt to use modes that meet those specifications (none of the built-in VESA modes does, so it'll probably bail if you don't have a modeline). The easiest thing to do is grab the VESA GTF based 640x480 modeline, half the pixel clock (the first number) and put "Interlace" at the end to get 640x480 interlaced, at ~15.25kHz.
If you REALLY want 640x240 for a game or something (that will be pretty much unusable as a desktop), it should accept that as well. You'll have to work out your own modeline, but the GTF should still give valid results. Just make sure that the hsync ends up being in the neighborhood of 15kHz and is in the range you specified. A 240 line (+some blanking/sync), 60Hz mode will need to be progressive to hit 15kHz.
Again, CHECK YOUR LOGS to see why it is invalidating your modeline. It usually very clearly tells you (hsync out of range, pixel clock too high/low, math doesn't work out, etc.).