]]]]]]]]]]]]]          MINDLESS MACHINES       [[[[[[[[[[[[[[[[[ 
              by Hubert Dreyfus and Stuart Dreyfus
       Computers Don't Think Like Experts, and Never Will
     (From The Sciences, November/December 1984, pp. 18-22)

            [Kindly uploaded by Freeman 10602PANC]

(Hubert  Dreyfus, author  of `What  Computers Can't  Do', teaches
philosophy  at the  University of  California at  Berkeley, where
Stuart Dreyfus is  a professor of  operations research.  They are
at work together on a book, `Putting Computers In Their Place'.)

   What  makes an  expert an  expert?  By  definition, he  of she
possesses  special  skill  in some  particular  field.   But what
qualities separate  the best in  that field from  the mediocre of
the merely very good?
   Among  the  first  to pose  this  question  was  Socrates, who
stalked Athens in search  of experts who could  reveal to him the
secrets of  their trades.  Plato  offers an  account of Socrates'
encounter   with   Euthyphro,   a   religious   prophet   and   a
self-proclaimed expert  at recognizing  piety.  ``I  want to know
what is characteristic of piety,''  Socrates put to him, ``to use
as a standard  whereby to judge  your actions and  those of other
men.''  But instead  of divulging a  set of guidelines, Euthyphro
did what every  other expert would do  when cornered by Socrates:
he gave examples from his own  area of expertise, in this case of
past  situations  in  which  men  and  gods  had  acted  piously.
Socrates  persisted  but  could  come  no  closer  to  a  general
principle.  In his later  dialogues, Plato suggested that perhaps
experts had once known the rules  by which they judge and act but
had simply forgotten them.   The philosopher's role, he believed,
was to help people rediscover those rules.
   Nearly twenty-five hundred  years later, Socrates' predicament
has  taken on  new relevance.   Computer scientists  have entered
into Socratic dialogues with experts  in many fields, in attempts
to glean their expertise and program it into machines.  While the
supreme goal of artificial  intelligence research continues to be
the design of computer programs that exhibit general intelligence
(like the  computer named Hal in Stanley Kubrick's  film `2001: A
Space  Odyssey'  [1968]),   many  computer   scientists,  calling
themselves  ``knowledge  engineers,''  have  taken  on  a  humble
challenge, that of designing programs  with expertise in a single
   Already expert systems  have been devised  that play checkers,
chess,  and  backgammon.   A  program  named  Prospector analyzes
drilling  maps  to  find mineral  deposits,  and  another, called
Dendral, examines  mass spectrograms  to determine  the molecular
structure of chemical compounds.  Mycin interprets the results of
blood  tests,  Puff  diagnoses  lung  disorders,  and Internist-I
offers complete medical checkups.   It would hardly be surprising
if, hidden away  in the basement of  one of the computer-software
houses, knowledge engineers  were at work  designing Euthyphro, a
program that tests for piety.
   The popular  press is rife  with accounts of  the successes of
these expert  systems and  with promises  of a  new generation of
computers  that will  possess  built-in expertise,  the so-called
``fifth generation.''  (The first four generations were marked by
the advent of vacuum tubes, transistors, integrated circuits, and
large-scale integrated circuits; the fifth revolution will be not
so much in  physical components as  in software.)  Expert systems
have  been the  subject  of cover  stories  in Business  Week and
Newsweek  and  of  the influential  book,  by  computer scientist
Edward Feigenbaum and science writer Pamela McCorduck, `The Fifth
Generation:   Artificial   Intelligence   and   Japan's  Computer
Challenge  to the  World'.   Feigenbaum's  own  research  and his
outspoken advocacy  of expert  systems have  propelled the United
States into a  competition (from which Feigenbaum,  as one of the
founders of the knowledge-engineering firm Teknowledge, stands to
profit   greatly)  to   be  the   first   nation  to   develop  a
fifth-generation  computer.  As  Feigenbaum and  McCorduck write:
``In the kind  of intelligent system  envisioned by the designers
of  the  Fifth Generation,  speed  and processing  power  will be
increased  dramatically; but  more  important, the  machines will
have  reasoning  power:  they  will  ...  learn,  associate, make
inferences, make decisions, and otherwise  behave in ways we have
always considered the exclusive province of human reason.''
   Will   expert  systems   duplicate,   or  even   surpass,  the
achievements  of  human  experts?  Fifteen  years  ago  it seemed
possible.  But not it seems that  all such attempts are doomed to
failure because of a  fundamental misunderstanding among computer
scientists as to how human experts operate.
   Knowledge engineers believe that once  the rules an expert has
acquired from years  of experience have been  pared down to their
essence,  a  computer, with  its  superior powers  of  logic, can
outperform  a  human  expert.    Human  experts,  Feigenbaum  and
McCormick  write,  ``build up  a  repertory of  working  rules of
thumb, or `heuristics,' that,  combined with book knowledge, make
them expert  practitioners.''  The task  of the  programmer is to
interrogate the experts and ``mine  those jewels of knowledge out
of their heads one by one.''
   The expert may have trouble  articulating the rules, but that,
to Feigenbaum and McCorduck, means only that he now automatically
follows  rules buried  deep within  his subconscious.   ``When we
learned how to tie  our shoes,'' they explain,  ``we had to think
very hard about the steps involved.  ... Now that we've tied many
shoes over our lifetime, that knowledge is `compiled,' to use the
computing  term  for  it;  it   no  longer  needs  our  conscious
attention.''  In their  Socratic view, the  rules are functioning
in the expert's mind whether he is conscious of them or not.  How
else, they ask, could we account for the fact that he has learned
to perform the task.
   For a  time, it  seemed that  some kinds  of experts,  such as
chess grand  masters, relied  so much  on reason  and memory that
their  tasks  could  readily be  accomplished  by  a high-powered
computer.  But we  are now convinced  that most expertise depends
on  unique  human  qualities  that can  never  be  mimicked  by a
machine.  No program using rules extracted from experts will ever
surpass those  experts.  The  stunning visions  of Feigenbaum and
his associates are, sadly, without basis.
   The    thirty-five-year   evolution    of    Arthur   Samuel's
checkers-playing computer  program suggests the  limits of expert
systems.   In 1947,  when  electronic computers  were  just being
developed, Samuel, then at the University of Illinois, decided to
write a  program to play  checkers.  After  calculating that even
the  fastest  computer,  if  it  were  to  generate  all possible
sequences  of  moves,  would   take  billions  upon  billions  of
centuries  just to  move the  first  piece, Samuel  changed tack.
Rather  than   build  a   machine  that   played  by  brute-force
calculation, he asked checkers-playing  masters (much as Socrates
asked Euthyphro) for rules of thumb  -- how to control the center
of the board, when to sacrifice  a piece for better position, and
so on -- that he could program into his computer.  The result was
a machine that followed the experts' rules but that still did not
play  like  a master.   So  Samuel  became the  first  (and today
remains one  of the  few artificial  intelligence researchers) to
develop  a  program  that  learned  form  its  own  mistakes.  He
programmed a computer to  vary how much weight  was given to each
rule of thumb and to retain  the combination of weights and rules
that worked best.   After playing many  games against itself, the
computer was able  to beat Samuel.   But it still  could not beat
the experts whose rules were the heart of the program.
   Samuel's checkers-playing  program is  one of  the best expert
systems ever built, but, even so,  its praises have been sung too
enthusiastically.  One often reads that the program plays at such
a high level  that only world champions  can beat it.  Feigenbaum
and  McCorduck, for  example,  report that  ``by  1961, [Samuel's
program]  played  championship  checkers,   and  it  learned  and
improved with  each game.''   In fact,  the program  did defeat a
state champion once and tied  a former world champion.  The world
champion, however,  ``turned around  and defeated  the program in
six  mail   games,''  as   Samuel  readily   admits.   And  after
thirty-five years  of effort, Samuel  says that  ``the program is
quite capable of  beating any amateur player  and can give better
players a good contest.''  But it is clearly no champion.
   A similar  fate has  befallen computers  that play  chess.  In
1968, the British international master  David Levy, who is also a
computer enthusiast (he  is chairman of  the London-based company
Intelligent Software), bet  five hundred pounds  that no computer
could defeat him in a match within the decade.  He won the bet by
beating the best program three-and-a-half games to one-and-a-half
games in  a best-of-six  match (a  draw rewards  each player with
half  a  game).   But  Levy  was  favorably  impressed  with  his
competititor and agreed  to take on the  computer again, in 1984.
As the rematch approached, a program called Cray Blitz, which had
just   won   the  world   computer-chess   championship   with  a
master-level score, was chosen as  Levy's opponent.  Not only did
Levy win decisively, four games  to none, but, more important, he
lost his long-held optimist  about chess-playing programs.  After
the match, he confessed to the `Los Angeles Times':
   During the last  few years I  had come to  believe more and
   more that it was possible for programs, within a decade, to
   play very strong grand master chess.  But having played the
   thing now, my feeling is  that a human world chess champion
   losing to a  computer program in  a serious match  is a lot
   further  away  than  I  thought.   Most  people  working on
   computer chess  are working  on the  wrong lines.   If more
   chess programmers studies the way human chess masters think
   and tried to emulate that to some extent, then I think they
   might get further.
   Despite the boasts of their creators, uncritically reported by
the  press,  all  other  expert   systems  have  also  failed  to
outperform their human counterparts.  One such inflated claim has
been  for  Prospector, which  Peter  E. Hart  and  his colleagues
devised  in   1980,  when  Hart   was  head   of  the  Artificial
Intelligence   Center,  at   the  Stanford   Research  Institute.
Prospector uses rules  elicited from expert  geologists to locate
mineral deposits.   Dan Rather reported on the `CBS Evening News'
that  SRI  researchers had  asked  Prospector to  find  the metal
molybdenum  and that  the  program had  then  directed them  to a
particular site  on Mount  Tolman, in  Washington, where,  lo and
behold, molybdenum was found.  In  reality, the program was given
information identifying  a field  of molydbenum  on Mount Tolman.
Prospector then mapped  out undrilled portions  of the field, and
subsequent drilling showed it to be basically correct about where
molybdenum did or  did not exist.   Unfortunately, the molybdenum
discovered was too deep to be worth mining.
   Unlike  Prospector,  Internist-I,  the  program  that performs
complete medical checkups, has  already been pitted against human
experts.   The  results  of  laboratory  tests  done  on nineteen
patients  were  given  to  Internist-I   and  to  two  groups  of
physicians:  a  team  of   clinicians  at  Massachusetts  General
Hospital,  in  Boston,  and   a  committee  of  medical  experts.
According to `The New England Journal of Medicine',of forty-three
diagnoses that  pathologists subsequently confirmed  (some of the
patients suffered  from several  afflictions), Internist-I missed
eighteen [42%], while the clinicians missed fifteen [35%] and the
medical experts  only eight [19%].   This same  story is repeated
again  and  again  -- in  Samuel's  checkers-playing  program, in
Prospector, and in  many other expert systems.   In each case the
computer can  do better than  beginners, but it  cannot rival the
very  experts whose  knowledge and  rules  it is  processing with
incredible speed an unerring accuracy.
   That so  many expert  systems have  reached the  same limit in
proficiency is more than coincidence.   The fault lies not in how
well computers  follow rules  but in  a mistaken  view of  how an
expert  acquires  expertise.   The  traditional  view  holds that
beginners  begin with  specific cases,  and  as they  become more
proficient they abstract and  develop more and more sophisticated
rules.   But a  better explanation  seems to  be that  skills are
acquired  in  just the  opposite  way, that  one  progresses from
abstract rules to specific  cases.  Even Feigenbaum and McCorduck
admit,  in  The Fifth  Generation,  that nearly  every  time they
suggest to an expert  the rule the expert  seems to be following,
they get a  Euthyphro-like response: `` `That's  true, but if you
see  enough patients/rocks/chit  design/instrument  readings, you
see  that  it  isn't  true after  all.'  ''   And  Feigenbaum and
McCorduck add with Socratic annoyance: ``At this point, knowledge
threatens to become ten thousand special cases.''
   This is  precisely what happens  in the acquisition  of one of
the  most  common expert  tasks,  driving a  car.   The beginning
driver is taught rules.  He may learn, for example, that when his
car is  traveling at  a speed of  twenty-five miles  per hour, he
should follow other  cars at a  distance of at  least two hundred
feet.  The driver checks his speed by reading the speedometer and
judges the distance between his car  and the next by counting car
   As  the  beginner  gains  experience,  he  makes  use  of less
objective cues.  He listens to his engine as well as looks at his
speedometer for cues about when  to shift gears.  He observes the
demeanor  as well  as the  position and  speed of  pedestrians to
anticipate  their  behavior.   And  he  learns  to  distinguish a
distracted or drunk driver from an alert one.  No number of rules
can   serve  as   real-life  examples   in  teaching   him  these
distinctions.   Engine sounds  cannot  be adequately  captured by
words, and no list of facts about a pedestrian at a crosswalk can
enable  a driver  to  predict his  behavior  as well  as  can the
experience of observing  people crossing streets  under a variety
of conditions.
   As the  driver acquires still  more experience,  the number of
factors  to  be  considered grows  rapidly.   To  cope  with this
explosion  of  information, he  gradually  adopts  a hierarchical
approach to  decision making.  Certain  considerations stand out,
while  others  recede  in  importance.   Without  the  burden  of
remembering  every  rule he  learned  when starting  out,  he now
drives less deliberately.   He follows cars  more closely, enters
traffic more daringly,  and occasionally violates  a speed limit.
He begins to  recognize the specifics  of a past  experience in a
present  one and  to  act accordingly.   By  the time  the driver
becomes proficient,  he will  be struck  by a  particular plan of
action  for  every   circumstance.   If,  for   instance,  he  is
approaching a curve on a rainy day,  he may sense that the car is
traveling  too  fast   and  lower  his   speed.   Still  he  must
consciously decide how to put the  plan into action -- whether to
release the accelerator or step on the brake.
   But the expert driver  is a breed apart:  he read the road and
adjusts  his  driving  without  any  conscious  awareness  of the
process.  The  car is  an extension  of his  body, and  he knows,
without thinking, when to slow down or shift lanes; his hands and
feet make the decision  for him, while his  mind may be a hundred
miles  away.  Intuition,  not reason,  governs his  actions.  The
rules that he once knew and used no longer guide his behavior.
   One could,  of course,  still argue  that despite appearances,
the  expert  driver's  brain  is  making  millions  of  rapid and
accurate inferences, just  like  a computer.   But the ability of
experts to recognize  the similarity between  a present situation
and one of tens of thousands of previously experienced situations
suggests that  the brain  does not  operate like  a heuristically
programmed computer.  Expertise seems more intuitive than we once
thought,  which would  explain  why philosophers  and programmers
from  Socrates to  Samuel have  had so  much trouble  getting the
expert to articulate the rules  he follows.  The expert is simply
not  following  rules.   He is  doing  just  what  Feigenbaum and
McCorduck  feared he  might  be doing:  recognizing  thousands of
special cases.
   That is why expert systems are never as good as experts.  When
an expert is asked for rules, he is effectively forced to regress
to the level  of a beginner  and recite rules  he no longer uses.
And when  those rules are  then incorporated into  a program, the
computer's  superior speed,  accuracy,  and memory  enable  it to
outdo human beginners applying the  same rules.  But no number of
rules and facts  can capture the  knowledge of an  expert who has
stored his experience of thousands of actual situations.
   Once  one relinquishes  the  assumption that  experts  must be
making  logical  inferences and  acknowledges  the  importance of
intuition, there  is no  reason to  believe that  a heuristically
programmed   computer  accurately   replicates   human  thinking.
Feigenbaum and McCorduck's  claim that ``we  have the opportunity
at this moment to do a new version of Diderot's `Encyclopedia', a
gathering up of all knowledge --  not just the academic kind, but
the informal,  experiential, heuristic  kind'' can  be seen  as a
late stage of Socratic thinking.   Those who proclaim the need to
begin  a  crash program  to  beat Japan  to  the fifth-generation
computer  can  be  seen as  false  prophets  blinded  by Socratic
assumptions  and ambition.   Euthyphro, the  expert on  piety who
offered Socrates  examples instead  of rules,  turns out  to have
been a true prophet after all.

                         *     *     *

Return to the ground floor of this tower
Return to the Main Courtyard
Return to Fort Freedom's home page