I have the same problem!

You are right about your putting success BUT only if you keep the ball on your left instep! Too much to the right, your putting will go sour in no time!! Another pointer- I had trouble with alignment to the hole and finally out of desperation, I physically stooped down behind the ball , put my finger on top of the center of the ball and aimed the putter to the hole- realized that my aiming was badly off even with the proper ball position! Then I took the stance at right angles to the putter face ( not to the shaft which could throw your aim off!) .

This is the same problem with the full swing too! You will be amazed how much your game can improve if you can aim properly and hit at the center of the ball! Look silly doing this? Not when you see your fellow players start commenting on your abilites to be consistently long and in the middle of the fairway! Again rememeber to stand at right angles to the club face and not to the shaft.

What I propose does not take any additional time- now it is a lot quicker because there is no second guessing on my shots once I am set up!! Just swing the d---n club!!

One additional caveat- make sure your left arm is fully extended ( this automatically takes care of the ground elevation/depression issues in the field!)