Recently had the same problem, but sorted it out. Turns out what I was doing was just lifting my arms, basically making a fake backswing. I would get caught in a bad position so the move down brought the hosel right into the ball.

I make sure to treat it the same as my full swing now, just stopping it with my left arm parallel to the ground, or 3/4 of the way, and swing nice and smooth....only difference is a narrower stance and my weight stays left. I still make sure to turn, not just lift. This helped me a lot and I've been pitching much better.