In general, the smoother the surface, the better the adhesion.
Primer is much better than bare wood because it fills in the porous spots in the wood, but I would recommend a sand and paint after priming, for best adhesion.
Don't paint with latex either.
Latex is rubbery, and will not stick as well as enamel.
If you use latex, you run the risk of the label sticking to the paint, but the paint coming off the primer.