Ok, number 1: It looks like you sucesfully moded your tv, however, have you tried another rgb source to see if your image is stable?
so that leads me to #2: Your picture looks like it lack sync, so you migh want to try another sync port, usually on rgb moded tuvs the yellow plug would do, but you can try another ports to see if your image gets stable.
#3, the way to set up correctly, is, by using an lcd monitor, the you install regular amd drivers first, then after reboot, you install calamity emudrivers, it will ask you to reboot, 2 times, once to anble drivers safety something like that, and reboot after you actually instal the drivers. once that happens and you open VMMaker and follow the instructions, basically you need to choose your conection and selec the dvi analog option to anable the drivers, from there you can conect you tv and should be able to see windows on your tv. Thats how ive done it several times.
Things to consider, double check you vga brekout cable, you should have one, if you are just using a regular cable, that wont work. By the way, what ind of cable are you using?
Ill help you get your stuff ready, just post pics of the cable you using, and will go from there.