Bottom line: Yes, it will be fine. (as long as the primer is dry)
in fact, oil based primer will form a better moisture barrier. The latex will adhere to it very well.
(my experience comes both from my own research and that I painted several apartments with both types of paint/primer when I was in property management)