you can get an inch or three by removing the door from it's hinges, and then removing the door's trim and casing from the wall. Usually outside doors come with a frame/casing around them, and then they are set in place, balanced and squared, and then trimmed in to make it match the rest of the wall.
It'll be a big pain in the ass, but you can get a few more inches this way.