Zinsser oil-based primers are quick-dry and sandable after 24 hours, from my experience. Sherwin Williams, Kelly-Moore, etc. (pro paint stores) sell oil primers that are not typically sanded after application.
Like JazBrown said, cure time is a major factor. Full curing time can take up to 6 months for some house primers!
Zinsser is sold at Home Depot and major paint stores. For oil-based get the Cover Stain and have it tinted to half the color of the top coat of paint (tinted gray for black paint, tinted pink for red paint, mint for green,etc)