I use 1000 grit between the last 2 coats, and 2500 final. I'll use 320 but very lightly initially.
Also, If you put 2 coats of an oil based paint, that will be more durable than 2 coats of primer, 3 coats of enamel spray paint, and clear coat. Enamel paints are just flaky without the perfect primer.
Durability: oil paint and a brush
Finish: primer, sandpaper, paint and clear coat.