I would sand to 220, not 320. 320 will not do much other than smooth out very minor imperfections. I assume that you are dealing with latex, in which case it won't fully cure for 3-6 weeks, so you are good to add another coat on top without sanding, at least as far as making it stick. I would only sand if the finish isn't smooth enough for your tastes, because the next coats will only emphasize the flaws.
There is some special sandpaper out there that is made for sanding latex paint, so when it starts to clog, which it will immediately, you can just flick it and the clogs will flake off the sandpaper. Most sandpapers will just be ruined after a little sanding on uncured latex paint. Also be super careful trying to sand latex that isn't fully cured, it will be soft enough to gouge very easily, and that will show through on the final coat.