I've always thought this about "1 or 2 club challenges" they don't hurt your score much at all, and the higher your handicap, the lower you'll score with less clubs.
Only players that can consistently hit long and staight with a driver are going to lose a few strokes.