Sounds like you are either coming on open or swinging from out to in.
Thats assuming that you have correct alignment and a good grip.

I'd check the grip and alignment first, easier than trying to change your swing.

You wont see it as much on the higher clubs as the more loft creates more backspin, which hides side-spin.
Thats why its worse with your driver.

If you can get yourself a lesson and save on heartache!