FEBRUARY 14, 2011
-
Read Later
READ LATERAvailable only to subscribers. SUBSCRIBE TODAY
-
Listen
ARTICLE AUDIO
- Font Size
In a series of three televised events starting tonight, an IBM supercomputer named Watson is set to compete against two human champions on the game show “Jeopardy!” If the computer wins—and in test matches that IBM has held, Watson has been dominating the humans—it will inspire comparisons to Deep Blue, the chess computer that shocked the world and prompted existential hand-wringing about the nature of human consciousness by beating Garry Kasparov in 1997.
But here’s the rub: “Jeopardy!”is actually a terrible way of proving that Watson is more intelligent than its opponents. A successful contest would certainly show that Watson can interpret complex sentences and recall esoteric information, and it would certainly show that IBM has designed a pretty smart computer. However, if the supercomputer triumphs, it will probably be for another reason entirely: because it can activate the buzzer most quickly.
This is how the “Jeopardy!” rules work: Whoever buzzes in first—using a clicking device usually compared to a large pen—gets the first chance at answering the question. The wrinkle, however, is that the contestants have to wait until Alex Trebek is completely finished reading the question before they are allowed to buzz in. Buzz too soon, and your buzzer is “locked out” for a quarter of a second, giving opponents the chance to jump in and answer before you. Contestants who wish to buzz in as fast as possible must either try to guess when the buzzers will activate (risking getting locked out if they are too early) or rely on their reflexes to buzz in when they see the lights (risking having reflexes that are too slow and allowing another player to buzz in first).
Mastery of this problem is critical to “Jeopardy!” success—possibly even more so than actual trivia knowledge. As Michael G. Dupree, who won “Jeopardy!”’s 1996 Tournament of Champions (TOC) and wrote How to Get on “Jeopardy!” and Win, put it in his treatise on the subject, “although each of the TOC competitors possessed an extremely broad range of knowledge, I came out on top because of my ability on the buzzer.” And Ken Jennings, who will be competing against Watson in the coming contest, has written on his website that, “‘Jeopardy!’ victory goes not to the biggest brain—it goes to the smoothest thumb. Timing on the tricky ‘Jeopardy!’ buzzer is often what separates the winner from the, well, non-winners, and the ‘Jeopardy!’ buzzer is a cruel mistress.”
That fact seems to have been borne out by Watson’s practice run. Of the 61 questions that can come up in a full game of “Jeopardy!” there is precisely one—the Final Jeopardy question—where we see how each of the contestants answers. On every other question, we only hear a second answer if the first one given is wrong. And in the recent test match between Watson, Jennings, and “Jeopardy!”champion Brad Rutter, none of the 15 questions were answered incorrectly by any of the players. In each case, the person who buzzed in first won the points associated with that clue. For all we know, all three players knew the answers to all the questions. Watson won that round, and it could easily have been because Watson was faster to the buzzer.
Indeed, when I called Watson’s creators to ask how the supercomputer controls its buzzer, they admitted that Watson does have a strong built-in advantage. According to David Shepler, who is IBM’s Challenge Program Manager for the Watson project, “The buzzer is enabled when the clue is done being read, when Alex Trebek gets to that last syllable, and the guy off stage pushes a button. That’s when people can buzz in, and at the same time a signal is sent to Watson saying the same thing—telling Watson that it can buzz in if it so desires.” This is akin to playing against an opponent with near-perfect reflexes. “When we built the demonstration system, the first incarnation of a fully functioning game-playing Watson, Watson was buzzing in electronically,” Shepler said, but they decided this was too unfair. IBM added an electromagnetic “hand” to Watson that will depress an actual buzzer which gives the humans a little more time, but not much. Asked if Watson still has an advantage under the new arrangement, Shepler told me that experienced players will sometimes get the jump on the supercomputer because they can anticipate the end of a clue, “but it's true that on average a machine has very good reflexes. The machine is probably going to beat a human at buzzing.” And the computer will never buzz in too early and get locked out.
So, while “Jeopardy!” clues provide a good test of Watson’s ability to decipher natural human language—and this ability really is quite amazing—the competition against the world’s best “Jeopardy!” players is probably a misleading gimmick. A better comparison with human minds might be an individual version of “Jeopardy!” where Watson answers as many questions correctly as it can, as quickly as it can, and then its score and time are compared to those of other trivia whizzes. And if Watson wins the televised match next week, don’t panic: It won’t mean that humanity’s superiority is in jeopardy.
Ezra Deutsch-Feldman is a web intern at The New Republic.
For more TNR, become a fan on Facebook and follow us on Twitter.
7 comments
When the NYTimes published an interactive Jeopardy! game against Watson last June, [http://www.nytimes.com/interactive/2010/06/16/magazine/watson-trivia-game.html?scp=8&sq=Watson%20jeopardy!&st=cse], beating it was pretty simple because there was no time pressure and the player could choose whether or not to answer a particular question. It has occurred to me that the game would be more fair if all buzzes within some short period (say 1/10 second - a standard estimate of the human reaction time) would be treated as tied and the opportunity to answer decided by a random choice.
- aduncanson
February 14, 2011 at 10:10am
TNR needs to quit having predictable articles contrarian articles so often. It gets tiresome and is just plain stupid at times (like this article). The article here is wrong on so many levels it's sad. The hype IS justified. That off-the-shelf hardware, along with clever software, can decipher spoken human language, which is rife with ambiguities and nuance is quite amazing. If you watched the Nova special last week about the design of Watson you'd have a deeper appreciation of the technological and engineering achievement this is. Yes, this IS a promo stunt and that is the point! Engineers and scientists too often have their work ignored or dismissed in the public eye. Much of it is the press is either too ignorant to understand it or the public finds it "boring". I for one applaud the public show here and wish to see more like it. And then we wonder why the US is losing it's technological edge. Articles like this point to why: our "scribes" aren't smart enough to help get the word out when we do things better than anyone else.
- tnmats
February 14, 2011 at 10:18am
The author here seems profoundly unacquainted with basic logic. Being fast to the button ma indeed be an advantage that makes Watson hard to beat, it's ability to answer questions being otherwise equal to the humans' - but then that's the amazing part, isn't it? Watson's ability to answer questions is, evidently, roughly equal to that of two well educated, highly successful players of a game requiring near instant recall of large amounts of anecdotal information cast in natural language form. If pushing a button fast was in an way sufficient to be in this game, then a photocell connected to a relay and stepping motor would have been all that is necessary. Make no mistake, stunt or not, Watson is a incredibly impressive advance in computing. Just compare it to Google for an instant. Ask Google a question an obscure question (or, actually, give it a word, since it doesn't actually parse language for questions) with a straightforward factual answer and it will give a million or so pages that MIGHT contain the answer to your question, if you had time look through them. Yes, the answer is more likely to be in the pages at the top of the list, but Google won't point it out for you. YOU have to parse all those words to find it. Watson will answer - might get it wrong sometimes, but it will answer the question, and although the Jeopardy stunt doesn't show you this, Watson could also tell you how it derived the answer - what sources it used, how it weighted them, etc. This is a huge advance in computers being sources of knowledge, not just computation. It's not yet the computer of science fiction wet dreams that converses with its human master providing instant answers on all of human knowledge on demand, but it's one hell of a big step in that direction.
- IowaBeauty
February 15, 2011 at 7:53am
1) It is an impressive statement of the state-of-the-art with respect to natural language processing and speed. 2) The natural language processing, however, should not be over-stated: it isn't sitting down to chat over tea with the two Jeopardy players (that would knock the socks off us!), it is answering a structured question ... every single time: structured question. 3) The data access shouldn't be over-impressive either - we've had high speed and high memory for awhile now (in technological years anyway) - brains hold vastly more data, using vastly less space and less energy. 4) At the end of the day Watson does the one and only thing that computers do, and it does nothing else: as we say in the business, it "pushes bits" ... in colloquial terms: it moves blocks. Specifically two blocks: "0" and "1". No block mover, no matter how fast, is doing something that is even in the same category as what a mind is doing. It doesn’t change matters that the blocks here happen to be electrons rather than wooden blocks. Wooden blocks don’t make minds, nor do electrons. 5) Yes brains, like computers, process data, but they obviously are doing something else too which manifests itself most profoundly in mammalian executive function. The "something else" is where the magic is. 6) To truly impress with a "huge advance" in AI, we must have a robot, the size of a bee, with a processor, the size of a bee's brain, which does all of the things a bee does, with the same (miniscule) amount of energy that a bee requires. When we have done that - we will have solved the big problems, and it will be time to be both awed and even a little afraid.
- dcwood10
February 15, 2011 at 12:54pm
Who knew? Thanks to Feldman's fine research, we can know how to interpret this event which I am sure will be infinitely hyped by IBM and the media. And he's only an "intern"!
- NR101900
February 15, 2011 at 2:26pm
"To truly impress with a "huge advance" in AI, we must have a robot, the size of a bee, with a processor, the size of a bee's brain, which does all of the things a bee does, with the same (miniscule) amount of energy that a bee requires. " Why this particular test? Would a bee be less intelligent if it where the size of a cow and used 75% of it's mass and energy to compute? No. Less adaptive, yes, but not less intelligent. I don't know whether to be amused or frustrated by those who say, in essense, that unless you can do what nature does in the same form factor, you've made no "huge advance." Advances are just that: advances. Newcomb's atmospheric engine was a huge advance in motive power, laughably inefficient and ponderous as it is by today's standard. Couldn't power a bumblebee, but led within a few decades to an engine that could power the industrial revolution. "At the end of the day Watson does the one and only thing that computers do, and it does nothing else: as we say in the business, it "pushes bits" ... in colloquial terms: it moves blocks. Specifically two blocks: "0" and "1". No block mover, no matter how fast, is doing something that is even in the same category as what a mind is doing. It doesn’t change matters that the blocks here happen to be electrons rather than wooden blocks. Wooden blocks don’t make minds, nor do electrons. Yes brains, like computers, process data, but they obviously are doing something else too which manifests itself most profoundly in mammalian executive function. The "something else" is where the magic is." OK, so this type of reductionism frustrates me even more. If all Watson does is manipulate bits, then "all" you or I do is convert C-H bonds to heat. Everything else is merely a side effect. Brains on the other hand are "obviously ... doing something else." Obvious in what way? By what stroke of analysis do you conclude that mammalian executive function is more than fuzzy processing of data followed by actuating electrochemical signals to move muscles as a result? I would say we don't know, but that there is substantive evidence that your executive function is in fact the interaction of multiple individually less robust and adaptive subsystems, and that a reasonable analysis suggests these too are amenable to similar decomposition, leading to the conclusion that data processing, albeit on a large scale, is in fact what is going on. It's not obvious that this is the case, but it sure as hell isn't obvious that it isn't.
- IowaBeauty
February 15, 2011 at 4:02pm
"The article here is wrong on so many levels it's sad. The hype IS justified. That off-the-shelf hardware, along with clever software, can decipher spoken human language, which is rife with ambiguities and nuance is quite amazing." The computer is fed a text-version of the question, from what I've heard.
- elopez
February 16, 2011 at 10:36am