You can only use your spoils if you have a set -- which consists of 3 spoils of the same color (red/blue/green), or a mixed set (1 red, 1 blue, 1 green). In the case where you had 4 and couldn't cash them in, you had two pair (two each of two colors). You're right, once you hit 5 spoils, you're forced to play them, because you're guaranteed to have one of those types of sets.
-- Major / 2243 points
-- May 21, 2015