Its probably a combination of lateral movement and casting. With me, the two seemed to go together, but I was a bigtime swayer early in my golfing life and had the same problem.