That's too many layers of primer! Better 1 layer of primer and 3 final ones. Let each layer of gloss paint harden well: 1 week at least (it sucks, I know, but you will be rewarded). Sand between every layer. Don't be a whimp on sanding, but don't remove everything either off course! It is even possible to end the painting with polishing to remove every dust particle.
If you like the gloss, keep it that way. Or are you sure the varnish is specially made for finishing that particular solid-gloss-paint? They really should be compatible.