from The Telegraph:
Computer passes the Turing Test? I'm not convinced
University of Reading claims a computer program passed the Turing Test and convinced judges it was a 13-year-old boy. But does it really count, asks Matthew Sparkes
Alan Turing Photo: AFP
By
Matthew Sparkes, Deputy Head of Technology
12:16PM BST 09 Jun 2014
133 Comments
“Computer passes Turing Test for the first time after convincing users it is human”,
read our headline over the weekend. Five different programs were put on trial at the Royal Society on Saturday and one – named Eugene Goostman – managed to convince the judges that it was a real boy.
The idea of testing for evidence of “thinking” in computers was proposed by Alan Turing in 1950, in what has become a seminal paper in the field of artificial intelligence. He called the test the “imitation game”: a judge talks to one human and one machine via a system that obscures everything but text. If he or she is fooled more than 30 per cent of the time, the machine can be thought of as intelligent.
Eugene fooled people 33 per cent of the time, which does pass Turing’s test, even if the vast majority of judges were able to see the truth.
But does this latest success – which coincidentally landed on the 60th anniversary of Turing’s death – count?
My first concern is that the code pretends to be a child of 13. This seems to bend the rules slightly. Turing does talk of creating a child in code in his paper, but only in the context of it being a step on the road to replicating an adult mind: create a child, which is in theory a simpler task, and then educate it as you would any real human.
Using the limited life experience, vocabulary and sophistication of language of a child is a slight smokescreen that could, perhaps, be used to mask a range of flaws or weak points.
Secondly, it pretends to be a 13-year-old child
from Ukraine, where the first language is not English. But the contest was performed in England with largely English-speaking judges. Again, this seems to be a slight fudge. It makes the test far easier if you have a plausible excuse for unnatural grammar.
Details of the recent test are thin on the ground – although they are being published in a scientific paper soon. The judges were made up of members of the public, Lord Sharkey, Robert Llewellyn (who played Kryten in Red Dwarf) and Aaron Sloman who is a world leading AI professor. There were a total of 300 conversations on the day. This all bodes well, and Professor John Barnden, formerly head of the British AI Society, adjudicated.
But I spoke to the University of Reading this morning to ask for full transcripts of the conversations and was denied. I was sent some previous transcripts from Eugene, one of which is replicated below, that do not exactly paint a picture of intelligence. It’s easy to see how 67 per cent of people became suspicious.
Perhaps the defining vote on this will fall to a $20,000 gambling slip: Ray Kurzweil, Google’s director of engineering, has a long-standing bet with Mitch Kapor, founder of the Electronic Frontier Foundation, on
whether any computer will pass the test by 2029. If the pessimistic Kapor pays out, I’ll concede that Eugene has passed muster. Until then, I’ll wait to read the published paper before deciding.
In any case, Eugene is no small achievement: creating code that can even come
close to natural language is a hugely difficult problem which seems simpler now than it did in 1950 purely because a huge number of talented people have been working on it since.
And the truth is that artificial intelligence is a science which already affects every one of us, every day, whether Eugene is a convincing schoolboy or not. In narrow fields computers are already beating humans at a range of tasks: expert systems can diagnose tricky diseases, computers are beating investment professionals at picking stocks and genetic algorithms design more efficient computer chips than humans can dream up – in many cases we don’t even understand how those designs work, but we go ahead and manufacture, ship and sell them regardless. They're in your mobile phone right now.
Transcript of Eugene talking to a human judge:
Judge: Hi, I'm Cathy.
Entity: Cathy! Nice to meet you! You're a woman, if I'm not mistaken.
Judge: Right. What is your name?
Entity: People call me Eugene. Or "Zhenya". It is a short name for Eugene in Russia and Ukraine.
Judge: Nice name. Are you from either area?
Entity: Agree
Oh, what a fruitful conversation;-)
Judge: Do you really think so?
Entity: No, your guess was wrong. I don't think so. By the way, I still don't know your specialty - or, possibly, I've missed it?
Judge: What speciality? What's yours?
Entity: Huh? And I forgot to ask you where you are from...
Judge: The universe and you?
Entity: Is it your hobby to ask little poor Jewish boys such tricky questions they don't have an answer for?
Computer passes the Turing Test? I'm not convinced - Telegraph