Singing Sock Puppets

A hot topic at the moment is gestural computing. Watch someone talking: not their mouth though - their hands. The hands will probably be in constant motion hammering home points, suggesting "maybe", saying "no". Even when we are speaking we use a lot more than sound to communicate. That is a reason why it is so easy for people to misunderstand and end up in flame wars with email. We automatically pick up lots of subtle signals about what a person's words actually mean from their posture, hands or expressions.

Gestural computing is all about communicating with computers more naturally. In particular it's about detecting the positions, movements and shapes of peoples' hands or arms and using that to control computers. Don't type, click quit or even speak to shut down a program just wave bye bye. In the film Minority Report Tom Cruise's character, John Anderton, uses gestural computing to explore the data from the pre-cogs: the human-computer interaction technique of the future?

A different twist on using hand gestures to communicate with computers is the virtual sock puppet. Sock puppets can communicate all sorts of expressions using different hand positions even if you aren't a ventriloquist and the sock is silent.

So what happens if you combine sock puppets and gestural computing? Could you get a computer to pick up the hand movements and actually make the sock speak...or even sing?

That is what Matthew Irvine Brown created for his MA at the Royal College of Art. His sock puppets have flex sensors that can detect the flexing of the mouth. By opening and closing the mouth the puppet sings up and down the scale. Fun yes, a bit silly maybe, but actually potentially really useful for teaching different kinds of scales (like blues, chromatic and pentatonic scales, for example) through play.

Teaching music aside, with a helping hand from technology, puppet shows may never be the same again...