Just like a hand saw if your blade enters the wood at anything other than straight it will end up drifting. Plus think blades bend pretty easily. It should get better with practice and a higher quality jigsaw, I use a Bosch.
However for long straight cuts I would highly suggest getting a circular saw. They are relatively innexpensive and will cut straighter and faster than a jigsaw. I only use my jigsaw for curves, large holes, and for cutting non-wood materials.