Set the crt desktop res to 320x240, load up hyperspin from the other monitor and see if the shaking is gone, if its still there it might be a problem with the chassis. I keep my crt in my cab at 320x240 because I dont like the shaking either. The only reason to run the crt in 640x480 or up is to be able to navigate and do work in windows, but since you are doing the work on a secondary monitor just leave the crt in a lower res progressive mode.
So I just checked it out and in game it does not shake at all, and as you thought, the res was 640x480. However I did not see any other option to select below that, that seemed to be the smallest one I could use and it also had the (Recommended) text next to it.
On another odd note, my RocketLauncherUI settings reverted and it was trying to find the RocketLauncherUI as the plugin and front end. Not sure why that happened, it was working fine last night. But after figuring that out I tested the games and the games weren't shaking like the UI was.
You most likely need to add that resolution to the user_modes-super.ini and generate the modelines again. It should look like this
## Desktop ##
640 x 480 @ 60.000000 desktop
320 x 240 @ 60.000000 desktop
<-------add new res here## Super resolutions ##
2560 x 240 @ 60.000000 super
2560 x 248 @ 60.000000 super
2560 x 256 @ 60.000000 super
2560 x 264 @ 60.000000 super
2560 x 272 @ 60.000000 super
2560 x 280 @ 60.000000 super
2560 x 288 @ 60.000000 super
2560 x 296 @ 60.000000 super
2560 x 304 @ 60.000000 super
2560 x 320 @ 60.000000 super
2560 x 336 @ 60.000000 super
2560 x 344 @ 60.000000 super
2560 x 352 @ 60.000000 super
2560 x 360 @ 60.000000 super
2560 x 368 @ 60.000000 super
2560 x 376 @ 60.000000 super
2560 x 384 @ 60.000000 super
2560 x 392 @ 60.000000 super
2560 x 400 @ 60.000000 super
2560 x 416 @ 60.000000 super
2560 x 432 @ 60.000000 super
2560 x 448 @ 60.000000 super
2560 x 464 @ 60.000000 super
2560 x 480 @ 60.000000 super
2560 x 496 @ 60.000000 super
2560 x 512 @ 59.000000 super
2560 x 544 @ 55.000000 super
2560 x 560 @ 54.000000 super
2560 x 768 @ 60.000000 super
2560 x 800 @ 60.000000 super
once you have that file saved open up vmmaker and click generate, then install the modes again. You should see 320x240 in the windows resolution settings now. As for rocketlauncher resetting its config doesnt surprise me, seems Hyperspin and RL are still full of bugs that have to be resolved.