Using the dark side

Shadows are strange. They're everywhere but usually we don't notice them. They just sit there, moored to the thing that casts them, ignored by us as we go about our daily lives. These seemingly useless dark patches in our visual world move around with us, change colour with the weather, hide things, and move with the light. Hannah Dee of the Grenoble Institute of Technology, in the French Alps tells us why she is bringing them into the limelight.

I'm a computer vision researcher, so I write programs that try to make sense of images and video. I'm really interested in shadows. I'm interested in how we can use shadows to help computers interpret images automatically, and how the human perceptual system uses shadows too.

The recipe for a shadow

If you’re going to teach computers how to use shadows, you’ve got to know why shadows look the way they do. A shadow has a simple recipe with just three ingredients – light, surface and object. When the object comes between the light and the surface the shadow is cast. The characteristics of those three ingredients determine how the shadow appears.

The sharpness of the edges of a shadow (the penumbra) depends upon how big the light source is: small lights cast sharp shadows, large lights cast fuzzy shadows. The shape of a shadow depends on the object that's casting it, and the shape of the surface that it's cast upon. If there are any other light sources you might get extra shadows, but if the other light is really fuzzy and spread out you won't. However, the colour of the fuzzy light will determine the colour of the shadows. The indirect light reflected off the blue sky explains why shadows seem bluish on a sunny day.

The way you move

That takes care of what a shadow looks like, but humans also get information from how shadows move. How a shadow moves depends on the movement of those three shadow ingredients, light, surface, and object, as well. Mostly, we think of shadows moving when the object moves (shadow puppets, for example) and we can use these moving shadows to tell us about the moving object. Sundials are an example of using a moving light source (the sun) to get a shadow to tell us about something in the world.

What does a light source see?

That’s not all the shadow tells you! The position and shape of a shadow gives us clues about the location of the object and its relation to the surface the shadow is cast upon. This is because the shadow is the area hidden from the light by the object. (Leonardo da Vinci knew this - and said "No luminous body ever sees the shadows that it generates", back in the 15th century.) We can say that the outline of the shadow is the outline of the object from the point of view of the light. That is, if you were where the light was, the silhouette of the object would be exactly the shape of the shadow, but you'd not be able to see the shadow because the object would be exactly in the way.

Because the shadow gives you a version of the world from the point of view of the light source, it's possible for a shadow to tell you about things you can't actually see. This can be used to terrifying effect in horror films, where the monster is creeping up out of shot, and the only evidence of their presence being the shadow cast on the wall.

What use are shadows?

Psychologists have spent a lot of time over the last ten years investigating how we see shadows, and what the human eyes and brain use them for. It seems that we use shadows for some things in our perception but not others. There's some evidence that the brain processes shadows very quickly, uses them to help work out the relationships between objects in the world, and then discards them. This is probably why we don't tend to notice them that much. Our brains use shadows a lot for working out where things are in depth – for 3D vision – and for tasks like grasping and picking up objects. We don't seem to use shadows for things that depend on small details, and we don't seem to use them to work out much about the surface or light source. Mostly we use shadows to tell us about the objects that are casting them. Humans never, except in optical illusions and tricks, mistake a shadow for an object.

A shadowy nemesis

Computer vision systems try and make sense of the world from analysing images and videos, and many people try to model computer vision systems on the human visual system. This seems a fairly sensible starting place, because it is one of the only examples we have that vision can work! Shadows, however, are problematic for computer vision systems. They keep getting mistaken for objects, which as we know is a mistake that humans do not make.

Computer vision researchers have been busy developing shadow detection and shadow removal techniques, so we can treat shadows as background and not as objects. How do you detect a shadow? Well, it's about the same colour (hue) as background, but a bit darker. So either you make your background detection system less fussy about dark stuff, or you explicitly detect shadows and then lump them in with the background. This gets most of the shadows, but not all, so researchers are constantly coming up with new ways to improve the detection. One example of an enhanced technique is based on noticing that, if the light stays the same, a piece of ground will look the same colour each time it’s in shadow. So if you can remember that you've got a head start in detecting the shadow.

The shadow robots

In computer vision and robotics, a few researchers are now beginning to exploit shadows as sources of information (not just as problematic dark patches to be ignored). Some Japanese researchers are fitting robots with bright lights so that the shadows they cast can tell them about the shape of the pipe or corridor they're going down. And I'm working in an international team trying to help a robot use shadows the way humans do. My Brazilian collaborators Paulo Santos and Valquiria Fenelon have a robot, and we're using shadows to help it navigate around its environment and to guess the relationship between the robot itself, objects around it, and a light source. Just like humans do when we watch a horror film and see the shadow of the monster...