Seeing color with three cones

Seeing color with three cones

We’ve come up with a bunch of different tools for describing lights, describing photopigments, and describing how a photopigment responds to a specific light. These tools are simple, but quite useful. They give us a way to describe a lot of things that can happen to light independent of the visual system, and also to describe what happens when light does encounter the visual system at the retina. Let’s start by talking about that first set of circumstances: How can we use emission and absorption spectra to model different things that can happen to light out in the world, before it gets to our eye? One good way to see how these lists are useful for describing light in the environment is to consider various ways that colors can be mixed together.

You almost certainly know a few things about mixing colors together. Suppose you scribble some red crayon a piece of paper and then scribble some blue crayon on top of that. I’m sure you can predict that the mixture of red and blue crayon will look sort of purple. We can do something similar with red and blue lights instead of red and blue crayons. Again, you may have guessed that this mixture of lights will also look purple. (Figure 1)

Figure 1 - Red crayon +Blue crayon = purple. Red light + Blue light = purple. So far, so good.

I want to make a small substitution to what we just did, though. First, let’s go ahead and scribble some red crayon on our piece of paper, but now I want to scribble some green crayon on top of that. What do you think this is going to look like? Honestly, it will probably look like a mess – this turns out sort of brown and gross. But suppose we did this with lights instead of crayons (like we did for red and blue). What will a mixture of red and green lights look like? It turns out that this doesn’t look like a mess at all – where the red and green lights overlap, you should see a rather nice yellow color (Figure 2).

Figure 2 - Red crayon + green crayon = sort of gross and brown. Red light + Green light = Yellow? 

Why are these two outcomes so different?

Why do these two ways of mixing light turn out differently? These two ways of mixing colors involve very different ways that we can combine distinct light spectra to get a new stimulus, and we can use our emission and absorption spectra to describe these concretely. Let’s do the easy one first: what’s going on when we mix lights together? This form of color mixing is called additive color mixing and it yielded this sort of surprising red+green=yellow outcome. The name ‘additive’ is appropriate because the way we combine two separate light spectra into a new spectra describing the mixture is incredibly simple: We can just add the values in the first spectrum to the values in the second one, and this gives us a new spectrum that contains the new combined intensity at each wavelength. This is a really basic computation, but it gives us something valuable: We can predict what a mixture of two lights will be if we know the spectra of those starting inputs. Once we know that, we can go on to consider what a photopigment would do when it encounters this new mixture of lights.

What about the situation when we’re mixing colors using crayons instead of lights? This kind of color mixing works quite differently, but we can model it with our lists too. We call this form of color mixing subtractive color mixing, which again is appropriate because what we’re really doing in these circumstances is selectively removing some light by mixing colors. The trick here is to consider the layer of crayon that we scribble on the paper as something that absorbs some light and lets other light through. That red crayon? We can think of it as a thin filter that let’s some light through to bounce off the white paper underneath, but stops other light in it’s tracks. Why does the crayon look red? Probably because that’s the wavelengths of light that it allows through. How do we model what’s happening to the light in this situation? We’ve already seen how we can use an absorption spectrum to describe what light a photopigment can absorb easily, and we can use something very similar to describe what the thin layer of crayon is doing to light. The red crayon must let reddish light (with long wavelengths) through, so it has small values at those wavelengths in its absorption spectrum. It absorbs other wavelengths of light, so it must have high values at those wavelengths.

If we multiply this absorption spectrum by the original light, we’ll get a new list that tells us how much light got soaked up by the crayon for each wavelength. But remember, this is the light that we don’t get to see! The crayon is ‘subtracting’ it from the original light by absorbing it. To find out what’s left, we can take the absorption spectrum and subtract each number in it from ‘1,’ which will tell us how much light at each wavelength passes through. If we multiply this list number-by-number with the light spectrum, we’ll end up with a new list that tells us what’s in the light that comes out the other side of the filter (Figure 3). Adding more colors means more filters that each only let some light through. Again, this final list we end up with is the tool we can use to start computing what a photopigment will do when it encounters this light. These lists let us describe several ways that lights of different color end up getting combined with one another due to their interactions with other lights or other kinds of material. The next step is to say more about calculating what happens next.

Figure 3 - Different filters absorb some wavelengths of light and allow others to pass through. By subtracting each value in the filters absorption spectrum from '1,' we end up with a list of fractions that tells us how much light at each wavelength made it through. This lets us describe the light on the other side of the filter in terms of its adjusted spectrum.

Besides these lists we can use to describe mixtures of light out in the world, we also  have a language for describing what a particular photopigment does when it encounters a particular light. By using a dot-product to combine the light spectrum describing the stimulus and the absorption spectrum describing the photopigment, we end up with a single number. That single number is how we’ll describe the signal produced by a photoreceptor with that photopigment – it’s the record your visual system has of what light was shining on it. Our next question is this: What exactly can you do with this? Perhaps more importantly, what can’t you do with this number?

Let’s explore this by keeping things simple: We’re going to imagine that we live in a world with a reduced set of possible wavelengths, which we process with photopigments with a reduced absorption spectrum. Specifically, we’ll call this world our “Rainbow World” because light will only come in 7 wavelengths corresponding to the colors of the rainbow: ROYGBIV = Red, Orange, Yellow, Green, Blue, Indigo, and Violet. This means that any light will be described by just 7 numbers. Our photopigments therefore only need 7 absorption coefficients to tell us how they interact with lights in the rainbow world, and we’ll use our dot products to compute the response of a photopigment with a light.

Figure 4 - In our toy "Rainbow World," there are only 7 kinds of light that can be seen.

We’ll start by inventing a single kind of photoreceptor with just one photopigment on it. What do we want to do with this thing? I’m going to argue that we’re interested in the following question: How good is this photoreceptor at telling different lights apart? That is, does the photoreceptor tend to respond differently to lights that have different ingredients (or different spectra, to be precise)? If this is our question, what kind of photopigment should we invent for our photoreceptor? I’m going to argue that we should make one that absorbs some light better than others, and has its peak right in the middle of our set of possible wavelengths. One reason to start here is that it turns out that these middle wavelengths are more common than shorter or longer wavelengths in natural environments, but we don’t really need this fact to continue. What matters is that we’ve invented a photopigment, which has the following absorption spectrum:

Pigment1 = [0.1 0.1 0.5 0.9 0.5 0.1 0.1];

Now that we have a photopigment, let’s feed it a light and compute a dot-product to get the response of the photoreceptor that has this pigment on it. Here’s an input that has a good bit of green/yellow stuff in it, and hardly any blue or violet wavelengths at all.

Light1 = [20 18 85 10 5 4 0].

If we pair up our two sets of numbers, multiply the pairs, and add them up we get the following:

Dot(Pigment1,Light1) = [2+1.8+42.5+9+2.5+.4]=58.2.

Remember, this value represents what the photoreceptor does in response to this input. If we want to know how useful this photoreceptor would be for telling different lights apart (by which we mean lights with different spectra), we need to figure out how likely it is that different lights will yield different responses. What I want to do now is point out a problem – a BIG problem – that affects using this photoreceptor to tell different lights apart. To see what the problem is, let’s consider a new light with a different spectrum:

Light2 = [0 0 84 8 0 0 90].

This light is a good bit different from Light1: The numbers in the list are obviously different, and they’re in different places. This light has a lot of violet light, for example, and very little green. We’d hope that our photoreceptor could tell these two lights apart, right? Let’s calculate our dot-product by pairing up these numbers with the absorption spectra above:

Dot(Pigment1, Light2) = [0+0+42+7.2+0+0+9]=58.2.

Oops.

This one photoreceptor can’t tell these two lights apart even though they’re different! This is kind of bad, but is it a deal-breaker? It turns out that it’s very much a deal-breaker, because the fact that this one photoreceptor can only give us one number to describe a light turns out to mean we can’t use it to make good guesses about the wavelengths that are present in an input. This is called the principle of univariance, and the reason it’s true is basically this: A light spectra has information about wavelength AND amplitude, that we collapse into one number by taking a dot-product with one photopigment. That means that the number a photoreceptor yields could be the result of smaller amounts of light at wavelengths that are absorbed well, or larger amounts of light at wavelengths that are absorbed poorly. It’s a little like knowing that the product of two numbers is 100, and trying to figure  out what the two numbers are. There are lots of possibilities, and we don’t have a good way to narrow them down.

So how do you tell lights with different spectra apart? You don’t just use one photopigment! Instead, you use three different photopigments on three different classes of cone photoreceptors – “Long,” “Medium,” and “Short” wavelength cones. You can see what their absorption spectra look like below:

Figure 5 - Absorption spectrum of the S-cones (blue), M-cones (green) and L-cones (red). The dark line shows you the absorption spectrum for the rods.

What does this allow us to do? Instead of just using one number to describe the retina’s response to a light, these three cones give us three numbers to work with, each computed with the same dot product operation. We can refer to a specific light with these “LMS-numbers,” and treat them like a new list of information that describes how the light was processed by the retina. Basically, even if one photopigment wouldn’t distinguish between two different lights, another one might.

What’s more, these three numbers also allow us to use a spatial framework for thinking about colors and your ability to tell them apart. Specifically, imagine that we constructed coordinate axes for each cone so that the x-axis was meant to reflect the response of the L-cones, the y-axis was meant to reflect the response of the M-cones, and the z-axis was meant to reflect the response of the S-cones. We could calculate the LMS numbers for any light that we were interested in, and then plot it along these axes. We’ll call this description of cone responses a cone space.

 Figure 6 - By plotting the response of one cone on the x-axis and the response of another on the y-axis, we have a 2D "cone space" that we can put cone responses into. All 3 cones would yield a 3D space with the same properties. 

Why would we do this? The reason this is useful is that it gives us a spatial framework for thinking about colors and how different they are from another: Distance in cone space should be related to perceived similarity. If colors are very close to one another in cone space, that means your retina did nearly the same thing in response to each color and you might have a hard time telling them apart. Alternatively, if they’re very far apart, your retina responded very differently to the two colors and you should find them easy to distinguish. The best part? We know how to calculate distances between points in 3D-space. If you don’t know how to do this, go take a look at the next post, which is called “Mathematical Preliminaries” to see how it works. Again, the point here is that we can calculate what your visual system does in response to a specific input and then make quantitative guesses about what you can see. It also turns out that some other things are easy to do in cone space – remember how we mixed colors additively earlier? This turns into something called vector addition in cone space, and really just involves adding arrows together to get a vector sum of colors that tells you what the result is (Figure 7). Calculating LMS numbers gives us a lot of nice ways to calculate what you should see in different circumstances.

Figure 7 - Instead of adding spectra, we can add responses in cone space through vector addition, which is just tip-to-tail combination of arrows.

OK, this is all pretty neat, but I do have to leave you with an important caveat: There are still colors we can’t tell apart with three cones. Even though we get three numbers to describe each light that your retina responds to, there is still plenty of room for physically different lights to yield the same retinal response. If that happens, we call those two colors metamers and you absolutely will not be able to tell them apart. More cones reduces the number of metamers that you have, but doesn’t eliminate them. Still, we’ve learned a lot about how photopigments allow you to respond to light, and how you use the responses of different photopigments in the cones to come up with a response that allows you tell some physically different lights apart. These differential responses are a version of wavelength encoding that is only available to your cones – remember, you only have one type of rod, so your rods can’t see color. This helps us understand some of the differences between photopic and scotopic vision that we observed earlier. We’re not just stuck with the fact that these two kinds of vision are different, we can calculate why they’re different.

Alright, if you need to, go review the math concepts in the next post. We’re going to continue to use models like this to try and understand different aspects of what you see, and things will get a little more complicated. We’ll be revisiting some of the same tools multiple times, however, so get familiar with them now if you’re feeling a little shaky about how to calculate things.