Doubtful it's the signal "shape" (meaning edge quality), especially since the LS and HC behaved the same. That was a long shot (but easy to test), anyway.
There are two major differences between those two circuits.
One is that it's logically different. Rather than XORing the syncs, it creates a multi-level OR of sorts then runs that through an inverter stage. The behavior will be quite different depending on the polarity of the input syncs (and if they're matched or not). In general, if the polarities are matched, you'll get "little" h-sync pulses during a frame, and "big" h-sync pulses during v-sync. This isn't really composite sync, but people do it all the time and, due to the way the sync separators on most analog monitors works, it often works. It can confuse some digital monitors, though. The non-linearity of Q1 will play with this some, so without simulating it I'm not entirely sure exactly what will come out the end.
The other big difference is the signal levels and source impedance. Your old converter put out 5Vpp from 1kohms. This one puts out ~2.5V from 70ohms. If you're still curious as to what's going on, this is easy enough to test. Replace the output resistor on your old circuit with a T-pad where every component has a value of 47ohms.
How are you hooking sync up to the commodore monitor? Does it have a dedicated sync input, or are you coupling the sync onto the video signal somehow eg. SoG?
Also, if you ever actually want to do an RGB to Composite/S-Video conversion, look up the AD725. It's much easier to use than that MC1377.
EDIT: Wait a sec, you say you're running it with the supply disconnected? What parts do and don't you have hooked up?