I can see the wood move away from the fence as I'm pushing it through - I have been pushing against the fence as well as pressing down to hold the piece flat to the table but it's like I'm being forced to allow the wood to drift towards the blade and away from the fence. I get the 1/32" measurement because that is approximately how much the cut deviates from the perpendicular line I draw on the piece.
So, based on what you've just said, I'm wondering if you really have a problem...
Forgive me is this is entirely too basic, but: the only thing a rip cut does is ensure that the edge of the wood against the fence is parallel to the edge cut by the saw blade. If the edge against the fence isn't straight, even this may not happen! If the line you've drawn isn't referenced on either the fence edge or the cut edge, then it's entirely possible that the
line isn't accurate!
I'd forget about the line and check to see if the fence is parallel to the blade. Check at both the front and back of the blade. If the fence and blade aren't parallel, adjust the fence so that it becomes parallel. To check the accuracy, grab a piece of scrap wood. Rip a bit of wood off one side, then flip the board 180 degrees (putting the edge you just cut against the fence) and rip a bit of wood off the opposite side. Then measure the width between the two cut edges at various points along the board. At this point, all the measurements should be identical. If they're not, you've still got a problem.
The other thing you might need to check for is deflection at the back of the fence. Perhaps you're pushing the wood against the fence too much and that's throwing it out of alignment. What type of saw are you using (benchtop, contractor's, cabinet)? What sort of fence does it have? Benchtop saws are notorious for having flimsy fences that deflect easily.