Ahh $%#@!
I first tried this on a square resolution smaller window. And the method works as expected. So I thought the polar-to-rect was having problems with non-square. So I made a large square window. I convert to polar, do stuff, then try to convert to rect, and BAM, not enough memory. I guess the 640MB is not good enough for the images I'm working with.
I guess I'll put off the artwork until I can get more memory, unless you know of ways to optimize photoshop.