There is a big difference usually in the length of a driver's shaft compared to the shaft of a 5 wood. This is really what creates the power. The arc of the swing is much longer with the driver and it takes more effort to swing the club because more leverage is created. Because of the of the length of the swing more things can also go wrong. A little breakdown in form can cause a big difference in flight path with a driver.

I think both schools of thought are correct. You may not realize the effort that you need to make to maintain club head speed with your driver. Go ahead and work on swing speed then. After a while though work on method to get consistency.