]]]]]]]]]]]]] MINDLESS MACHINES [[[[[[[[[[[[[[[[[ by Hubert Dreyfus and Stuart Dreyfus Computers Don't Think Like Experts, and Never Will (From The Sciences, November/December 1984, pp. 18-22) [Kindly uploaded by Freeman 10602PANC] (Hubert Dreyfus, author of `What Computers Can't Do', teaches philosophy at the University of California at Berkeley, where Stuart Dreyfus is a professor of operations research. They are at work together on a book, `Putting Computers In Their Place'.) What makes an expert an expert? By definition, he of she possesses special skill in some particular field. But what qualities separate the best in that field from the mediocre of the merely very good? Among the first to pose this question was Socrates, who stalked Athens in search of experts who could reveal to him the secrets of their trades. Plato offers an account of Socrates' encounter with Euthyphro, a religious prophet and a self-proclaimed expert at recognizing piety. ``I want to know what is characteristic of piety,'' Socrates put to him, ``to use as a standard whereby to judge your actions and those of other men.'' But instead of divulging a set of guidelines, Euthyphro did what every other expert would do when cornered by Socrates: he gave examples from his own area of expertise, in this case of past situations in which men and gods had acted piously. Socrates persisted but could come no closer to a general principle. In his later dialogues, Plato suggested that perhaps experts had once known the rules by which they judge and act but had simply forgotten them. The philosopher's role, he believed, was to help people rediscover those rules. Nearly twenty-five hundred years later, Socrates' predicament has taken on new relevance. Computer scientists have entered into Socratic dialogues with experts in many fields, in attempts to glean their expertise and program it into machines. While the supreme goal of artificial intelligence research continues to be the design of computer programs that exhibit general intelligence (like the computer named Hal in Stanley Kubrick's film `2001: A Space Odyssey' [1968]), many computer scientists, calling themselves ``knowledge engineers,'' have taken on a humble challenge, that of designing programs with expertise in a single field. Already expert systems have been devised that play checkers, chess, and backgammon. A program named Prospector analyzes drilling maps to find mineral deposits, and another, called Dendral, examines mass spectrograms to determine the molecular structure of chemical compounds. Mycin interprets the results of blood tests, Puff diagnoses lung disorders, and Internist-I offers complete medical checkups. It would hardly be surprising if, hidden away in the basement of one of the computer-software houses, knowledge engineers were at work designing Euthyphro, a program that tests for piety. The popular press is rife with accounts of the successes of these expert systems and with promises of a new generation of computers that will possess built-in expertise, the so-called ``fifth generation.'' (The first four generations were marked by the advent of vacuum tubes, transistors, integrated circuits, and large-scale integrated circuits; the fifth revolution will be not so much in physical components as in software.) Expert systems have been the subject of cover stories in Business Week and Newsweek and of the influential book, by computer scientist Edward Feigenbaum and science writer Pamela McCorduck, `The Fifth Generation: Artificial Intelligence and Japan's Computer Challenge to the World'. Feigenbaum's own research and his outspoken advocacy of expert systems have propelled the United States into a competition (from which Feigenbaum, as one of the founders of the knowledge-engineering firm Teknowledge, stands to profit greatly) to be the first nation to develop a fifth-generation computer. As Feigenbaum and McCorduck write: ``In the kind of intelligent system envisioned by the designers of the Fifth Generation, speed and processing power will be increased dramatically; but more important, the machines will have reasoning power: they will ... learn, associate, make inferences, make decisions, and otherwise behave in ways we have always considered the exclusive province of human reason.'' Will expert systems duplicate, or even surpass, the achievements of human experts? Fifteen years ago it seemed possible. But not it seems that all such attempts are doomed to failure because of a fundamental misunderstanding among computer scientists as to how human experts operate. Knowledge engineers believe that once the rules an expert has acquired from years of experience have been pared down to their essence, a computer, with its superior powers of logic, can outperform a human expert. Human experts, Feigenbaum and McCormick write, ``build up a repertory of working rules of thumb, or `heuristics,' that, combined with book knowledge, make them expert practitioners.'' The task of the programmer is to interrogate the experts and ``mine those jewels of knowledge out of their heads one by one.'' The expert may have trouble articulating the rules, but that, to Feigenbaum and McCorduck, means only that he now automatically follows rules buried deep within his subconscious. ``When we learned how to tie our shoes,'' they explain, ``we had to think very hard about the steps involved. ... Now that we've tied many shoes over our lifetime, that knowledge is `compiled,' to use the computing term for it; it no longer needs our conscious attention.'' In their Socratic view, the rules are functioning in the expert's mind whether he is conscious of them or not. How else, they ask, could we account for the fact that he has learned to perform the task. For a time, it seemed that some kinds of experts, such as chess grand masters, relied so much on reason and memory that their tasks could readily be accomplished by a high-powered computer. But we are now convinced that most expertise depends on unique human qualities that can never be mimicked by a machine. No program using rules extracted from experts will ever surpass those experts. The stunning visions of Feigenbaum and his associates are, sadly, without basis. The thirty-five-year evolution of Arthur Samuel's checkers-playing computer program suggests the limits of expert systems. In 1947, when electronic computers were just being developed, Samuel, then at the University of Illinois, decided to write a program to play checkers. After calculating that even the fastest computer, if it were to generate all possible sequences of moves, would take billions upon billions of centuries just to move the first piece, Samuel changed tack. Rather than build a machine that played by brute-force calculation, he asked checkers-playing masters (much as Socrates asked Euthyphro) for rules of thumb -- how to control the center of the board, when to sacrifice a piece for better position, and so on -- that he could program into his computer. The result was a machine that followed the experts' rules but that still did not play like a master. So Samuel became the first (and today remains one of the few artificial intelligence researchers) to develop a program that learned form its own mistakes. He programmed a computer to vary how much weight was given to each rule of thumb and to retain the combination of weights and rules that worked best. After playing many games against itself, the computer was able to beat Samuel. But it still could not beat the experts whose rules were the heart of the program. Samuel's checkers-playing program is one of the best expert systems ever built, but, even so, its praises have been sung too enthusiastically. One often reads that the program plays at such a high level that only world champions can beat it. Feigenbaum and McCorduck, for example, report that ``by 1961, [Samuel's program] played championship checkers, and it learned and improved with each game.'' In fact, the program did defeat a state champion once and tied a former world champion. The world champion, however, ``turned around and defeated the program in six mail games,'' as Samuel readily admits. And after thirty-five years of effort, Samuel says that ``the program is quite capable of beating any amateur player and can give better players a good contest.'' But it is clearly no champion. A similar fate has befallen computers that play chess. In 1968, the British international master David Levy, who is also a computer enthusiast (he is chairman of the London-based company Intelligent Software), bet five hundred pounds that no computer could defeat him in a match within the decade. He won the bet by beating the best program three-and-a-half games to one-and-a-half games in a best-of-six match (a draw rewards each player with half a game). But Levy was favorably impressed with his competititor and agreed to take on the computer again, in 1984. As the rematch approached, a program called Cray Blitz, which had just won the world computer-chess championship with a master-level score, was chosen as Levy's opponent. Not only did Levy win decisively, four games to none, but, more important, he lost his long-held optimist about chess-playing programs. After the match, he confessed to the `Los Angeles Times': During the last few years I had come to believe more and more that it was possible for programs, within a decade, to play very strong grand master chess. But having played the thing now, my feeling is that a human world chess champion losing to a computer program in a serious match is a lot further away than I thought. Most people working on computer chess are working on the wrong lines. If more chess programmers studies the way human chess masters think and tried to emulate that to some extent, then I think they might get further. Despite the boasts of their creators, uncritically reported by the press, all other expert systems have also failed to outperform their human counterparts. One such inflated claim has been for Prospector, which Peter E. Hart and his colleagues devised in 1980, when Hart was head of the Artificial Intelligence Center, at the Stanford Research Institute. Prospector uses rules elicited from expert geologists to locate mineral deposits. Dan Rather reported on the `CBS Evening News' that SRI researchers had asked Prospector to find the metal molybdenum and that the program had then directed them to a particular site on Mount Tolman, in Washington, where, lo and behold, molybdenum was found. In reality, the program was given information identifying a field of molydbenum on Mount Tolman. Prospector then mapped out undrilled portions of the field, and subsequent drilling showed it to be basically correct about where molybdenum did or did not exist. Unfortunately, the molybdenum discovered was too deep to be worth mining. Unlike Prospector, Internist-I, the program that performs complete medical checkups, has already been pitted against human experts. The results of laboratory tests done on nineteen patients were given to Internist-I and to two groups of physicians: a team of clinicians at Massachusetts General Hospital, in Boston, and a committee of medical experts. According to `The New England Journal of Medicine',of forty-three diagnoses that pathologists subsequently confirmed (some of the patients suffered from several afflictions), Internist-I missed eighteen [42%], while the clinicians missed fifteen [35%] and the medical experts only eight [19%]. This same story is repeated again and again -- in Samuel's checkers-playing program, in Prospector, and in many other expert systems. In each case the computer can do better than beginners, but it cannot rival the very experts whose knowledge and rules it is processing with incredible speed an unerring accuracy. That so many expert systems have reached the same limit in proficiency is more than coincidence. The fault lies not in how well computers follow rules but in a mistaken view of how an expert acquires expertise. The traditional view holds that beginners begin with specific cases, and as they become more proficient they abstract and develop more and more sophisticated rules. But a better explanation seems to be that skills are acquired in just the opposite way, that one progresses from abstract rules to specific cases. Even Feigenbaum and McCorduck admit, in The Fifth Generation, that nearly every time they suggest to an expert the rule the expert seems to be following, they get a Euthyphro-like response: `` `That's true, but if you see enough patients/rocks/chit design/instrument readings, you see that it isn't true after all.' '' And Feigenbaum and McCorduck add with Socratic annoyance: ``At this point, knowledge threatens to become ten thousand special cases.'' This is precisely what happens in the acquisition of one of the most common expert tasks, driving a car. The beginning driver is taught rules. He may learn, for example, that when his car is traveling at a speed of twenty-five miles per hour, he should follow other cars at a distance of at least two hundred feet. The driver checks his speed by reading the speedometer and judges the distance between his car and the next by counting car lengths. As the beginner gains experience, he makes use of less objective cues. He listens to his engine as well as looks at his speedometer for cues about when to shift gears. He observes the demeanor as well as the position and speed of pedestrians to anticipate their behavior. And he learns to distinguish a distracted or drunk driver from an alert one. No number of rules can serve as real-life examples in teaching him these distinctions. Engine sounds cannot be adequately captured by words, and no list of facts about a pedestrian at a crosswalk can enable a driver to predict his behavior as well as can the experience of observing people crossing streets under a variety of conditions. As the driver acquires still more experience, the number of factors to be considered grows rapidly. To cope with this explosion of information, he gradually adopts a hierarchical approach to decision making. Certain considerations stand out, while others recede in importance. Without the burden of remembering every rule he learned when starting out, he now drives less deliberately. He follows cars more closely, enters traffic more daringly, and occasionally violates a speed limit. He begins to recognize the specifics of a past experience in a present one and to act accordingly. By the time the driver becomes proficient, he will be struck by a particular plan of action for every circumstance. If, for instance, he is approaching a curve on a rainy day, he may sense that the car is traveling too fast and lower his speed. Still he must consciously decide how to put the plan into action -- whether to release the accelerator or step on the brake. But the expert driver is a breed apart: he read the road and adjusts his driving without any conscious awareness of the process. The car is an extension of his body, and he knows, without thinking, when to slow down or shift lanes; his hands and feet make the decision for him, while his mind may be a hundred miles away. Intuition, not reason, governs his actions. The rules that he once knew and used no longer guide his behavior. One could, of course, still argue that despite appearances, the expert driver's brain is making millions of rapid and accurate inferences, just like a computer. But the ability of experts to recognize the similarity between a present situation and one of tens of thousands of previously experienced situations suggests that the brain does not operate like a heuristically programmed computer. Expertise seems more intuitive than we once thought, which would explain why philosophers and programmers from Socrates to Samuel have had so much trouble getting the expert to articulate the rules he follows. The expert is simply not following rules. He is doing just what Feigenbaum and McCorduck feared he might be doing: recognizing thousands of special cases. That is why expert systems are never as good as experts. When an expert is asked for rules, he is effectively forced to regress to the level of a beginner and recite rules he no longer uses. And when those rules are then incorporated into a program, the computer's superior speed, accuracy, and memory enable it to outdo human beginners applying the same rules. But no number of rules and facts can capture the knowledge of an expert who has stored his experience of thousands of actual situations. Once one relinquishes the assumption that experts must be making logical inferences and acknowledges the importance of intuition, there is no reason to believe that a heuristically programmed computer accurately replicates human thinking. Feigenbaum and McCorduck's claim that ``we have the opportunity at this moment to do a new version of Diderot's `Encyclopedia', a gathering up of all knowledge -- not just the academic kind, but the informal, experiential, heuristic kind'' can be seen as a late stage of Socratic thinking. Those who proclaim the need to begin a crash program to beat Japan to the fifth-generation computer can be seen as false prophets blinded by Socratic assumptions and ambition. Euthyphro, the expert on piety who offered Socrates examples instead of rules, turns out to have been a true prophet after all. * * *
Return to the ground floor of this tower
Return to the Main Courtyard
Return to Fort Freedom's home page