In theory, it should work fine. In practice, it depends.
Try it and find out.
If the server was a high reliability design, you might have to contend with registered RAM and/or ECC. I'm reasonably certain you can't mix and match. It's one of those things I know there's a perfectly good reason why, I just can't remember off hand.
Then you get into latency issues. Google around for a bit if you encounter latency mismatching.