I find the same thing happens on my backswing, which makes the ball appear to shift a little, breaking my focus. I take this as a sign that I'm over swinging. Lately, I've been trying to get a feel for only swinging back as far as a I can and still have both eyes on the ball. It feels like I'm only taking a partial swing, but the results are great.

-Clay