You can only use your spoils if you have a set -- which consists of 3 spoils of the same color (red/blue/green), or a mixed set (1 red, 1 blue, 1 green). In the case where you had 4 and couldn't cash them in, you had two pair (two each of two colors). You're right, once you hit 5 spoils, you're forced to play them, because you're guaranteed to have one of those types of sets.
Highest rank/rating -- Major / 2225 points -- May15, 2014
Kuma's GC Qualifier: City Mogul
Jules Verne Contest - Team Finals
The Great Community Map Project: Tournament Five
Serbia's NFL 2013 Poly Trip