The Science of Seeing

The Science of Seeing

Our goal in this series of lessons is to try and understand how you use your visual sense to understand the world around you. I can’t do better than David Marr, who said that vision was really all about knowing “what was where by looking.” This is what we want to figure out: How do you do that? How does your vision work?

This may seem like an odd question to ask. Your vision feels effortless. Look around you and there’s the world: People you can identify, objects you can recognize, colors, surfaces, textures, all of which you quickly grok without any trouble. What is there to understand?

Here’s something to consider: Look at the picture below and see if you can tell the difference between the Dalmatians and the pictures of chocolate-chip ice cream. If you want to try and do this under time pressure, you can follow the link below the picture to visit sporcle.com and actually get a score. My guess is that you’ll do pretty well.

 

Figure 1 - Visit https://www.sporcle.com/games/sproutcm/dalmatian_not_choc_chip to see if you actually know what a pile of dogs looks like.

But how did you do that? Imagine you have to try and describe to an alien (who knows nothing about dogs or dessert) how to do this task. What’s the trick? My guess is that you might say something like, “Look for eyes.” or “If you see legs and a body, then it’s a Dalmatian.” But both of these kinds of statements push the question down the road without really helping! How do you know something is an eye, a leg, or a body? “Look for fur.” isn’t good advice if you can’t tell me what fur is and what it looks like. So what exactly are you doing when you choose which images are which?

You’re doing something with patterns of light. You’re sensing them (by which we mean something like measuring or encoding them) and you’re perceiving them (by which we mean interpreting what they imply about objects and surfaces in the world). You’re using your eyes and a good bit of your brain to do it. You’re also leveraging a lot of different processes that bridge the gap between a real Dalmatian out in the world and your experience of seeing that Dalmatian – processes that include inferring color, edges, surfaces, volumes, materials, and category labels from patterns of light in a cluttered, complex, and changing world.

This is a lot to understand, but we have two specific things going for us: (1) Compared to other sciences, we get to do experiments by just looking at things. Our visual experiences are our data! (2) We have many quantitative models of human vision that help us account for why we see what we do. (3) We have a range of tools for examining the anatomy of the visual system. We’re going to use all three of these approaches to try and explain how you sense and perceive the world around you using your vision. Along the way, we’ll end up talking a little about physics, a little about physiology, and a good bit about computation, all in service of characterizing what vision is and how it works.

Compared to other descriptions of human vision, this discussion is going to be based as much as possible on observations that are easy to make with accessible materials. At times we’ll have to compromise on this, but for the most part we’ll always try to appeal to your observations to ground what we’re doing in your own visual sense. In large part, this will be formalized in the lab exercises referenced throughout these posts. You can find supporting documents for these exercises at the links provided with each post, but you’ll have to find some of your own materials like lasers, lenses, scissors, etc. Some of these things won’t be too hard to find, but others may be trickier. I’ve included a list of materials and a few links to helpful vendors in case you want to get your own stuff and try these things out.

Besides observation, this discussion is also going to involve describing how to use specific computations to model what the visual system is doing. If you’re not a fan of math, I make no apologies for this: Understanding vision, really understanding it, means understanding how these models work, how to use them to make predictions, and understanding their limits. I believe it’s not enough to say “I get the concepts…” without being able to do some work calculating how specific mechanisms work. Luckily, we’ll stay pretty close to addition, multiplication, and just a tiny bit of trigonometry. The tricky part is understanding why we’re doing the mathematics that we’re doing.

With that, we’re ready to get started! We’ll begin by considering the very first part of seeing: Having something to see in the first place. That is, we’ll begin by talking about light.