Either you play the game by the rules or you don't. The correct procedure in this case is to declare the ball lost and call the following group to play through while you return to the tee to play your 5th shot.

The penalty for a lost ball is stroke and distance. While it goes against my grain and is positively illegal in competetive golf, people playing social golf in the circumstances you describe sometimes do what you did. The reason your partner said a two shot penalty is that the penalty for a lost ball is stroke AND distance. As you did not go back to where you played your previous shot he added a shot for that distance plus the penalty shot ie TWO strokes.

Neither of you were right but your partner was more right than you were.