I found out some interesting things. These buttons just toggle through the list of available modes. For example, if device_video_adjust is set to 'generate_yclock', it's on 'auto', and there is only one mode - the one generated. If it's set to 'none', then it will automatically select a number of modelines available within the bounds of the device_video_format, and hence you can toggle through them, back and forth (the list doesn't roll over).
To test something, I inserted a modeline at display_video, in this case ntsc_320x240. This was then the only modeline available. (I'm not sure why one would want to do this, necessarily, even if using for only one game, when the auto mode is more effective, but...) I noticed the default sync was negative, but I tried it (again) anyways, as well as changing to positive sync. There was no change in the image - neither on this monitor nor the multisync (which displayed either sync polarity fine) I was using to access windows and pre-test things.
I went back to the doc and looked again at what it says about using composite sync:
If the monitor accepts a composite sync signal, instead of using a sync conversion circuits you can also try twisting the two H and V VGA signal together. It works if you select VGA negative H and V sync on the programs. To be on safe side I DO NOT RECOMMEND THIS HACK to connect sync signals directly together. Technically you should never just tie sync signal lines together. They are not usually designed for this, so this can damage your video card. If you try, use with caution.
So maybe I need a sync converter? I didn't want to have to pull the monitor out and run separate sync lines, but it's looking like I'll have to. Anyways, thank you for your assistance.