You can only use your spoils if you have a set -- which consists of 3 spoils of the same color (red/blue/green), or a mixed set (1 red, 1 blue, 1 green). In the case where you had 4 and couldn't cash them in, you had two pair (two each of two colors). You're right, once you hit 5 spoils, you're forced to play them, because you're guaranteed to have one of those types of sets.
-- Major / 2220 points
-- Oct 22, 2012
Tournament wins:Kuma's GC Qualifier: City Mogul
Jules Verne Contest - Team Finals
The Great Community Map Project: Tournament Five