If you mix yellow and cyan pigments and then illuminate it with white light, the yellow pigment subtracts all light except for the red, yellow and green, and the cyan pigment subtracts all light except for the yellow, green and cyan. What you get left with is green, and as a result, mixing yellow and cyan subtractively yields green.
In order to express this using RGB colour values, you would start with white, subtract the opposites of yellow and cyan, which are blue and red, and then you get green. Alternatively, if the values are 0 to 1, you can also imagine it as multiplying the RGB channels of the yellow and cyan together, as only the light that both colours of pigment let through is visible. (These two ways to think about it give different results, and the multiplicative way is more accurate to real life, but both agree on the basics of subtractive colour mixing.)
I've been experimenting, and I think I've found a better way to mix two colours. Square the components of both colours before averaging them, and then take the square root of the result.
Here is the result of mixing a bright red with a bright lime, using the "regular" method:
And here is how it looks if you square and square root:
It looks a lot better. The reason this works is complicated, and relates to how computers store colour values perceptually rather than linearly, but this is close to how the two colours would blur together in real life, such as with an out of focus camera.
For inspiration for a good set of colour operators, I would look at 3D vector operations. Our perception of colours inherently has three degrees of freedom, whether they be red, green and blue, cyan, magenta and yellow, or hue, saturation and brightness, and mathematics has plenty of ideas on how to operate on three-dimensional values like that.
