it may be as it was for me that you are square and nicely set up at adress, but as to take the club away from the ball you may be doing what slips into my game occasionly that you are fanning the face open slightly without realising.

i cure this by keeping face square to taget of first portion of takeaway for longer then going ahead with normal swing.

my reckoning, for me is if set up / grip is a little out i will get a little fade / draw but if i fan open/ closed disaster will follow.