Employ the Best or Bust!

Queen Mary University of London

Why is it important that algorithms can explain themselves?

Imagine you run a hi-tech and very cool, Scandanavian company that develops software. You are about to expand to other countries and need to employ lots more people. The trouble is your company is so cool that everyone wants to work for you. You get a million applications for every job. As your employees work with children it’s really important they are honest, so you decide to create a system to spot dishonest people, making it easier to vet applicants. You want an algorithmic version of the Precogs in the film Minority Report who can spot criminals before they commit crimes! You get a bunch of mugshots of people in prison from across the world to train the system on. You also get pictures of people who you know definitely aren’t criminals as you vetted them: the employee pictures of everyone who has worked for your company. Your program comes up with a way to make decisions about who is dishonest and who isn’t, successfully identifying most of the prisoners in a second set of photos you originally held back, so you happily start to use it. It automatically rejects applicants from their photos without a human ever having to look at them. It seems to work, filtering out large numbers of applicants. A success! What you don’t realise is the pattern the algorithm spotted was nothing to do with being dishonest. People who worked for your company, based in Scandinavia where people tend to have light-coloured hair, are blonder overall than people in the rest of the world. It is rejecting people based on their hair colour! You never find that out though as it never explains itself so you never realise that, while employing some brilliant blondes, you are rejecting lots of the most creative (and honest) people in the world who just happen to have darker hair. Ten years later, out-competed by other companies with brilliant multi-cultural teams your company goes bust.