My hands are always further away from me in the downswing than at address. I have to align the toe of the club at the ball at address to hit the sweet spot on the way back. I have attributed it to the fact that I am slightly more squat in the downswing. Not sure if this is hand sway though.