Return to the ground floor of this tower
Return to the Main Courtyard
Return to Fort Freedom's home page
]]]]]]]]]]]]] MINDLESS MACHINES [[[[[[[[[[[[[[[[[
by Hubert Dreyfus and Stuart Dreyfus
Computers Don't Think Like Experts, and Never Will
(From The Sciences, November/December 1984, pp. 18-22)
[Kindly uploaded by Freeman 10602PANC]
(Hubert Dreyfus, author of `What Computers Can't Do', teaches
philosophy at the University of California at Berkeley, where
Stuart Dreyfus is a professor of operations research. They are
at work together on a book, `Putting Computers In Their Place'.)
What makes an expert an expert? By definition, he of she
possesses special skill in some particular field. But what
qualities separate the best in that field from the mediocre of
the merely very good?
Among the first to pose this question was Socrates, who
stalked Athens in search of experts who could reveal to him the
secrets of their trades. Plato offers an account of Socrates'
encounter with Euthyphro, a religious prophet and a
self-proclaimed expert at recognizing piety. ``I want to know
what is characteristic of piety,'' Socrates put to him, ``to use
as a standard whereby to judge your actions and those of other
men.'' But instead of divulging a set of guidelines, Euthyphro
did what every other expert would do when cornered by Socrates:
he gave examples from his own area of expertise, in this case of
past situations in which men and gods had acted piously.
Socrates persisted but could come no closer to a general
principle. In his later dialogues, Plato suggested that perhaps
experts had once known the rules by which they judge and act but
had simply forgotten them. The philosopher's role, he believed,
was to help people rediscover those rules.
Nearly twenty-five hundred years later, Socrates' predicament
has taken on new relevance. Computer scientists have entered
into Socratic dialogues with experts in many fields, in attempts
to glean their expertise and program it into machines. While the
supreme goal of artificial intelligence research continues to be
the design of computer programs that exhibit general intelligence
(like the computer named Hal in Stanley Kubrick's film `2001: A
Space Odyssey' [1968]), many computer scientists, calling
themselves ``knowledge engineers,'' have taken on a humble
challenge, that of designing programs with expertise in a single
field.
Already expert systems have been devised that play checkers,
chess, and backgammon. A program named Prospector analyzes
drilling maps to find mineral deposits, and another, called
Dendral, examines mass spectrograms to determine the molecular
structure of chemical compounds. Mycin interprets the results of
blood tests, Puff diagnoses lung disorders, and Internist-I
offers complete medical checkups. It would hardly be surprising
if, hidden away in the basement of one of the computer-software
houses, knowledge engineers were at work designing Euthyphro, a
program that tests for piety.
The popular press is rife with accounts of the successes of
these expert systems and with promises of a new generation of
computers that will possess built-in expertise, the so-called
``fifth generation.'' (The first four generations were marked by
the advent of vacuum tubes, transistors, integrated circuits, and
large-scale integrated circuits; the fifth revolution will be not
so much in physical components as in software.) Expert systems
have been the subject of cover stories in Business Week and
Newsweek and of the influential book, by computer scientist
Edward Feigenbaum and science writer Pamela McCorduck, `The Fifth
Generation: Artificial Intelligence and Japan's Computer
Challenge to the World'. Feigenbaum's own research and his
outspoken advocacy of expert systems have propelled the United
States into a competition (from which Feigenbaum, as one of the
founders of the knowledge-engineering firm Teknowledge, stands to
profit greatly) to be the first nation to develop a
fifth-generation computer. As Feigenbaum and McCorduck write:
``In the kind of intelligent system envisioned by the designers
of the Fifth Generation, speed and processing power will be
increased dramatically; but more important, the machines will
have reasoning power: they will ... learn, associate, make
inferences, make decisions, and otherwise behave in ways we have
always considered the exclusive province of human reason.''
Will expert systems duplicate, or even surpass, the
achievements of human experts? Fifteen years ago it seemed
possible. But not it seems that all such attempts are doomed to
failure because of a fundamental misunderstanding among computer
scientists as to how human experts operate.
Knowledge engineers believe that once the rules an expert has
acquired from years of experience have been pared down to their
essence, a computer, with its superior powers of logic, can
outperform a human expert. Human experts, Feigenbaum and
McCormick write, ``build up a repertory of working rules of
thumb, or `heuristics,' that, combined with book knowledge, make
them expert practitioners.'' The task of the programmer is to
interrogate the experts and ``mine those jewels of knowledge out
of their heads one by one.''
The expert may have trouble articulating the rules, but that,
to Feigenbaum and McCorduck, means only that he now automatically
follows rules buried deep within his subconscious. ``When we
learned how to tie our shoes,'' they explain, ``we had to think
very hard about the steps involved. ... Now that we've tied many
shoes over our lifetime, that knowledge is `compiled,' to use the
computing term for it; it no longer needs our conscious
attention.'' In their Socratic view, the rules are functioning
in the expert's mind whether he is conscious of them or not. How
else, they ask, could we account for the fact that he has learned
to perform the task.
For a time, it seemed that some kinds of experts, such as
chess grand masters, relied so much on reason and memory that
their tasks could readily be accomplished by a high-powered
computer. But we are now convinced that most expertise depends
on unique human qualities that can never be mimicked by a
machine. No program using rules extracted from experts will ever
surpass those experts. The stunning visions of Feigenbaum and
his associates are, sadly, without basis.
The thirty-five-year evolution of Arthur Samuel's
checkers-playing computer program suggests the limits of expert
systems. In 1947, when electronic computers were just being
developed, Samuel, then at the University of Illinois, decided to
write a program to play checkers. After calculating that even
the fastest computer, if it were to generate all possible
sequences of moves, would take billions upon billions of
centuries just to move the first piece, Samuel changed tack.
Rather than build a machine that played by brute-force
calculation, he asked checkers-playing masters (much as Socrates
asked Euthyphro) for rules of thumb -- how to control the center
of the board, when to sacrifice a piece for better position, and
so on -- that he could program into his computer. The result was
a machine that followed the experts' rules but that still did not
play like a master. So Samuel became the first (and today
remains one of the few artificial intelligence researchers) to
develop a program that learned form its own mistakes. He
programmed a computer to vary how much weight was given to each
rule of thumb and to retain the combination of weights and rules
that worked best. After playing many games against itself, the
computer was able to beat Samuel. But it still could not beat
the experts whose rules were the heart of the program.
Samuel's checkers-playing program is one of the best expert
systems ever built, but, even so, its praises have been sung too
enthusiastically. One often reads that the program plays at such
a high level that only world champions can beat it. Feigenbaum
and McCorduck, for example, report that ``by 1961, [Samuel's
program] played championship checkers, and it learned and
improved with each game.'' In fact, the program did defeat a
state champion once and tied a former world champion. The world
champion, however, ``turned around and defeated the program in
six mail games,'' as Samuel readily admits. And after
thirty-five years of effort, Samuel says that ``the program is
quite capable of beating any amateur player and can give better
players a good contest.'' But it is clearly no champion.
A similar fate has befallen computers that play chess. In
1968, the British international master David Levy, who is also a
computer enthusiast (he is chairman of the London-based company
Intelligent Software), bet five hundred pounds that no computer
could defeat him in a match within the decade. He won the bet by
beating the best program three-and-a-half games to one-and-a-half
games in a best-of-six match (a draw rewards each player with
half a game). But Levy was favorably impressed with his
competititor and agreed to take on the computer again, in 1984.
As the rematch approached, a program called Cray Blitz, which had
just won the world computer-chess championship with a
master-level score, was chosen as Levy's opponent. Not only did
Levy win decisively, four games to none, but, more important, he
lost his long-held optimist about chess-playing programs. After
the match, he confessed to the `Los Angeles Times':
During the last few years I had come to believe more and
more that it was possible for programs, within a decade, to
play very strong grand master chess. But having played the
thing now, my feeling is that a human world chess champion
losing to a computer program in a serious match is a lot
further away than I thought. Most people working on
computer chess are working on the wrong lines. If more
chess programmers studies the way human chess masters think
and tried to emulate that to some extent, then I think they
might get further.
Despite the boasts of their creators, uncritically reported by
the press, all other expert systems have also failed to
outperform their human counterparts. One such inflated claim has
been for Prospector, which Peter E. Hart and his colleagues
devised in 1980, when Hart was head of the Artificial
Intelligence Center, at the Stanford Research Institute.
Prospector uses rules elicited from expert geologists to locate
mineral deposits. Dan Rather reported on the `CBS Evening News'
that SRI researchers had asked Prospector to find the metal
molybdenum and that the program had then directed them to a
particular site on Mount Tolman, in Washington, where, lo and
behold, molybdenum was found. In reality, the program was given
information identifying a field of molydbenum on Mount Tolman.
Prospector then mapped out undrilled portions of the field, and
subsequent drilling showed it to be basically correct about where
molybdenum did or did not exist. Unfortunately, the molybdenum
discovered was too deep to be worth mining.
Unlike Prospector, Internist-I, the program that performs
complete medical checkups, has already been pitted against human
experts. The results of laboratory tests done on nineteen
patients were given to Internist-I and to two groups of
physicians: a team of clinicians at Massachusetts General
Hospital, in Boston, and a committee of medical experts.
According to `The New England Journal of Medicine',of forty-three
diagnoses that pathologists subsequently confirmed (some of the
patients suffered from several afflictions), Internist-I missed
eighteen [42%], while the clinicians missed fifteen [35%] and the
medical experts only eight [19%]. This same story is repeated
again and again -- in Samuel's checkers-playing program, in
Prospector, and in many other expert systems. In each case the
computer can do better than beginners, but it cannot rival the
very experts whose knowledge and rules it is processing with
incredible speed an unerring accuracy.
That so many expert systems have reached the same limit in
proficiency is more than coincidence. The fault lies not in how
well computers follow rules but in a mistaken view of how an
expert acquires expertise. The traditional view holds that
beginners begin with specific cases, and as they become more
proficient they abstract and develop more and more sophisticated
rules. But a better explanation seems to be that skills are
acquired in just the opposite way, that one progresses from
abstract rules to specific cases. Even Feigenbaum and McCorduck
admit, in The Fifth Generation, that nearly every time they
suggest to an expert the rule the expert seems to be following,
they get a Euthyphro-like response: `` `That's true, but if you
see enough patients/rocks/chit design/instrument readings, you
see that it isn't true after all.' '' And Feigenbaum and
McCorduck add with Socratic annoyance: ``At this point, knowledge
threatens to become ten thousand special cases.''
This is precisely what happens in the acquisition of one of
the most common expert tasks, driving a car. The beginning
driver is taught rules. He may learn, for example, that when his
car is traveling at a speed of twenty-five miles per hour, he
should follow other cars at a distance of at least two hundred
feet. The driver checks his speed by reading the speedometer and
judges the distance between his car and the next by counting car
lengths.
As the beginner gains experience, he makes use of less
objective cues. He listens to his engine as well as looks at his
speedometer for cues about when to shift gears. He observes the
demeanor as well as the position and speed of pedestrians to
anticipate their behavior. And he learns to distinguish a
distracted or drunk driver from an alert one. No number of rules
can serve as real-life examples in teaching him these
distinctions. Engine sounds cannot be adequately captured by
words, and no list of facts about a pedestrian at a crosswalk can
enable a driver to predict his behavior as well as can the
experience of observing people crossing streets under a variety
of conditions.
As the driver acquires still more experience, the number of
factors to be considered grows rapidly. To cope with this
explosion of information, he gradually adopts a hierarchical
approach to decision making. Certain considerations stand out,
while others recede in importance. Without the burden of
remembering every rule he learned when starting out, he now
drives less deliberately. He follows cars more closely, enters
traffic more daringly, and occasionally violates a speed limit.
He begins to recognize the specifics of a past experience in a
present one and to act accordingly. By the time the driver
becomes proficient, he will be struck by a particular plan of
action for every circumstance. If, for instance, he is
approaching a curve on a rainy day, he may sense that the car is
traveling too fast and lower his speed. Still he must
consciously decide how to put the plan into action -- whether to
release the accelerator or step on the brake.
But the expert driver is a breed apart: he read the road and
adjusts his driving without any conscious awareness of the
process. The car is an extension of his body, and he knows,
without thinking, when to slow down or shift lanes; his hands and
feet make the decision for him, while his mind may be a hundred
miles away. Intuition, not reason, governs his actions. The
rules that he once knew and used no longer guide his behavior.
One could, of course, still argue that despite appearances,
the expert driver's brain is making millions of rapid and
accurate inferences, just like a computer. But the ability of
experts to recognize the similarity between a present situation
and one of tens of thousands of previously experienced situations
suggests that the brain does not operate like a heuristically
programmed computer. Expertise seems more intuitive than we once
thought, which would explain why philosophers and programmers
from Socrates to Samuel have had so much trouble getting the
expert to articulate the rules he follows. The expert is simply
not following rules. He is doing just what Feigenbaum and
McCorduck feared he might be doing: recognizing thousands of
special cases.
That is why expert systems are never as good as experts. When
an expert is asked for rules, he is effectively forced to regress
to the level of a beginner and recite rules he no longer uses.
And when those rules are then incorporated into a program, the
computer's superior speed, accuracy, and memory enable it to
outdo human beginners applying the same rules. But no number of
rules and facts can capture the knowledge of an expert who has
stored his experience of thousands of actual situations.
Once one relinquishes the assumption that experts must be
making logical inferences and acknowledges the importance of
intuition, there is no reason to believe that a heuristically
programmed computer accurately replicates human thinking.
Feigenbaum and McCorduck's claim that ``we have the opportunity
at this moment to do a new version of Diderot's `Encyclopedia', a
gathering up of all knowledge -- not just the academic kind, but
the informal, experiential, heuristic kind'' can be seen as a
late stage of Socratic thinking. Those who proclaim the need to
begin a crash program to beat Japan to the fifth-generation
computer can be seen as false prophets blinded by Socratic
assumptions and ambition. Euthyphro, the expert on piety who
offered Socrates examples instead of rules, turns out to have
been a true prophet after all.
* * *
Return to the ground floor of this tower
Return to the Main Courtyard
Return to Fort Freedom's home page