You cant become too strong without major weight work, so dont worry about getting too buff. Train with some of those big rubber bands, they increase strength and flexibilty quite nicely and are simple to use.

I train in Karate and I have found that it helped my golf a good bit. You might want to try something like that as an alternative.

D.