I was thinking about this the other night...
In the Panorama version of Race Drivin.. you can hook up multiple monitors. According to Jeff Anderson of Videotopia.. you can assign each monitor their own 3d coordinates... and I believe any perspective angles. I don't believe there is any limits to the number of monitors you can pop on there... He theorized you could make a sphere of them. Hes shown it working with a total of 5 monitors (4 side + original display).
What if you set up a 2nd camera to be correct 3d camera distance from the main display perspective.. and then used a half silvered mirror with 2nd display.. and or 3d display tech with 3d glasses... ? That could be incredible!

You could take it even further, making each side monitor in 3d as well.