Phrase Detectives: Puzzling with a purpose

People often learn by playing games. Now computers can too – but we still get to play the games not them! Jon Chamberlain of Essex University tells us more.

People spend a lot of time playing computer games: around 200 million hours a day are spent playing them in the US alone. Most games are intended for entertainment, but what if the things you did within a game could help computers learn to do things in the real world better? Perhaps by playing word puzzles you could train search engines to find documents more relevant to your searches, or by tagging images you could show computers how to identify things in pictures.

Games with a Purpose

This is the idea behind "GWAP" computer games ("Games with a Purpose"), pioneered by Luis von Ahn's team at Carnegie Mellon University in 2003. The first GWAP was the ESP Game (now called Google Image Labeller). In the ESP Game players are assigned a partner and they have to guess the same word tag for a displayed image. Lots of people played the game, allowing it to rapidly build up a huge database of tags for images. Those tags are now being used to improve image searching (guess why Google was interested!).

Phrase Detectives

A big problem for computers is they don't really understand the words in the documents they store. They can search for matching words easily and return the sentences they were in. That is enough for search engines to find web pages that might be about the topic we searched for. It would be so much better if they could do things like summarise the documents they found, though. Then you wouldn't have to go to the web page itself to get an idea if it was the sort of thing you were looking for, or read the whole thing if it was. People find that kind of summarising hard to do, so how on earth can a dumb computer be expected to understand what is written well enough? That's where the Essex University GWAP game comes in.

They have created a game called "Phrase Detectives" that is being used to create a large resource of computer-friendly text that would otherwise be too time-consuming and expensive to produce. People playing the game add certain clues to the passages they read, and those clues that are just the kind of thing that computers need to take a step towards understanding text. They tell the computer which words in passages are linked.

In particular, the Phase Detectives game is collecting data on what are known as anaphoric references. They are just references to things that have been mentioned earlier. Take the sentence: "Jon went to the shop and he purchased a newspaper". The word "he" is an anaphoric reference. "Jon" is the thing that it refers to. It’s hard for computers to work out what this kind of reference is talking about automatically. Most of the time we find it easy though.

That's where the Phrase Detectives comes in. Players are given passages to read and have to pair up the references. It uses gaming techniques to make working out these references engaging and playable. Points scores and leaderboards are used to motivate players to make high-quality decisions. The players are also rated against known answers and have their annotations timed. That helps indicate how hard tasks are and how good the player is at the tasks in general.

How can you be sure that the player's decisions really are good ones though? On top of scoring points for completing tasks, each player's decisions are judged by other players and rewarded if they agree. That motivates everyone to concentrate on giving the best answers, as well as adding an important element of social interaction.

The ultimate idea is that future computer programs will be able to use all the data collected in the game about what refers to what in passages. It will be used by computers to automatically learn how to work out references themselves in new text they come across.

Social networking

The next phase of Phrase Detectives is to integrate with Facebook. A team from Modul University in Vienna developed a successful Facebook GWAP called "US08 Sentiment Quiz". Players have to indicate if a given phrase has a positive or negative sentiment about candidates in the 2008 US presidential elections. By combining the power of existing friendship networks with collaborative scoring, GWAPs like Phrase Detectives can become addictive social games.

Sack the experts?

With enough volunteers completing tasks can this kind of mass intelligence eventually come to the same answer as a skilled expert would? It is in answering this question that GWAP interfaces really prove their worth. Mass volunteer collaboration has already proven to be a great way to complete difficult tasks. Wikipedia, which attracts millions of contributors adding and editing encyclopaedia entries, is the obvious example.

The results from Phrase Detectives will be analysed at the end of 2008 to see whether all those players are producing good clues that could be used to improve anaphor resolution systems. In the meantime, if you enjoy word puzzles, give your puzzling a purpose. Go play Phrase Detectives and help teach a computer to puzzle.