I only return cards that are in competition, but I always return a competition card.
If you are off 14 and scored a 43 or so then you will only get dropped to 11 or 12, its not like they will cut you to 6 or anything.