ملاحظات
الفصل الأول: ماذا لو نجحنا؟
(1)
The first edition of my textbook on AI, co-authored with Peter Norvig, currently director
of
research at Google: Stuart Russell and Peter Norvig, Artificial Intelligence: A
Modern Approach, 1st ed. (Prentice Hall, 1995).
(2)
Robinson developed the resolution algorithm, which
can, given enough time, prove any logical consequence of a set of first-order logical
assertions.
Unlike previous algorithms, it did not require conversion to propositional logic.
J. Alan Robinson, “A
machine-oriented logic based on the resolution principle,” Journal of the
ACM 12 (1965): 23–41.
(3)
Arthur Samuel, an American pioneer of the computer era, did his early work at IBM.
The
paper describing his work on checkers was the first to use the term machine
learning, although Alan Turing had already talked about “a machine that can learn from
experience” as early as 1947. Arthur Samuel, “Some studies in machine learning using
the game of
checkers,” IBM Journal of Research and Development 3 (1959):
210–29.
(4)
The “Lighthill Report,” as it became known, led to the termination of research funding
for AI except at the universities of Edinburgh and Sussex: Michael James Lighthill,
“Artificial
intelligence: A general survey,” in Artificial Intelligence: A Paper
Symposium (Science Research Council of Great Britain, 1973).
(5)
The CDC 6600 filled an entire room and cost the equivalent of $20 million. For its
era it was incredibly powerful, albeit a million times less powerful than an
iPhone.
(6)
Following Deep Blue’s victory over Kasparov, at least one commentator predicted that
it
would take one hundred years before the same thing happened in Go: George Johnson,
“To test a powerful
computer, play an ancient game,” The New York Times, July 29,
1997.
(7)
For a highly readable history of the development of nuclear technology, see Richard
Rhodes, The Making of the Atomic Bomb (Simon & Schuster,
1987).
(8)
A simple supervised learning algorithm may not have this effect, unless it is wrapped
within an A/B testing framework (as is common in online marketing settings). Bandit
algorithms and
reinforcement learning algorithms will have this effect if they operate with an explicit
representation of user state or an implicit representation in terms of the history
of interactions
with the user.
(9)
Some have argued that profit-maximizing corporations are already out-of-control
artificial entities. See, for example, Charles Stross, “Dude, you broke the future!”
(keynote, 34th
Chaos Communications Congress, 2017). See also Ted Chiang, “Silicon Valley is turning
into its own
worst fear,” Buzzfeed, December 18, 2017. The idea is explored
further by Daniel Hillis, “The first machine intelligences,” in Possible
Minds: Twenty-Five Ways of Looking at AI, ed. John Brockman (Penguin Press,
2019).
(10)
For its time, Wiener’s paper was a rare exception to the prevailing view that all
technological progress was a good thing: Norbert Wiener, “Some moral and technical
consequences of
automation,” Science 131 (1960): 1355–58.
الفصل الثاني: مفهوم الذكاء في البشر والآلات
(1)
Santiago Ramon y Cajal proposed synaptic changes as the site of learning in 1894,
but
it was not until the late 1960s that this hypothesis was confirmed experimentally.
See Timothy
Bliss and Terje Lomo, “Long-lasting potentiation of synaptic transmission in the dentate
area of
the anaesthetized rabbit following stimulation of the perforant path,” Journal of Physiology 232 (1973): 331–56.
(2)
For a brief introduction, see James Gorman, “Learning how little we know about the
brain,” The New York Times, November 10, 2014. See also Tom
Siegfried, “There’s a long way to go in understanding the brain,” ScienceNews, July 25, 2017. A special 2017 issue of the journal Neuron (vol. 94, pp. 933–1040) provides a good overview of many
different approaches to understanding the brain.
(3)
The presence or absence of consciousness — actual subjective experience — certainly
makes a difference in our moral consideration for machines. If ever we gain enough
understanding
to design conscious machines or to detect that we have done so, we would face many
important moral
issues for which we are largely unprepared.
(4)
The following paper was among the first to make a clear connection between
reinforcement learning algorithms and neurophysiological recordings: Wolfram Schultz,
Peter Dayan,
and P. Read Montague, “A neural substrate of prediction and reward,” Science 275 (1997): 1593–99.
(5)
Studies of intracranial stimulation were carried out with the hope of finding cures
for various mental illnesses. See, for example, Robert Heath, “Electrical self-stimulation
of the
brain in man,” American Journal of Psychiatry 120 (1963):
571–77.
(6)
An example of a species that may be facing self-extinction via addiction: Bryson
Voirin, “Biology and conservation of the pygmy sloth, Bradypus
pygmaeus,” Journal of Mammalogy 96 (2015):
703–7.
(7)
The Baldwin effect in evolution is usually
attributed to the following paper: James Baldwin, “A new factor in evolution,” American Naturalist 30 (1896): 441–51.
(8)
The core idea of the Baldwin effect also appears in the following work: Conwy Lloyd
Morgan, Habit and Instinct (Edward Arnold,
1896).
(9)
A modern analysis and computer implementation demonstrating the Baldwin effect:
Geoffrey Hinton and Steven Nowlan, “How learning can guide evolution,” Complex Systems 1 (1987): 495–502.
(10)
Further elucidation of the Baldwin effect by a computer model that includes the
evolution of the internal reward-signaling circuitry: David Ackley and Michael Littman,
“Interactions between learning and evolution,” in Artificial Life
II, ed. Christopher Langton et al. (Addison-Wesley, 1991).
(11)
Here I am pointing to the roots of our present-day concept of intelligence, rather
than describing the ancient Greek concept of nous, which had a
variety of related meanings.
(12)
The quotation is taken from Aristotle, Nicomachean
Ethics, Book III, 3, 1112b.
(13)
Cardano, one of the first European mathematicians to consider negative numbers,
developed an early mathematical treatment of probability in games. He died in 1576,
eighty-seven
years before his work appeared in print: Gerolamo Cardano, Liber de ludo
aleae (Lyons, 1663).
(14)
Arnauld’s work, initially published anonymously, is often called The Port-Royal Logic: Antoine Arnauld, La
logique, ou l’art de penser (Chez Charles Savreux, 1662). See also Blaise Pascal,
Pensées (Chez Guillaume Desprez,
1670).
(15)
The concept of utility: Daniel Bernoulli, “Specimen theoriae novae de mensura
sortis,” Proceedings of the St. Petersburg Imperial Academy of
Sciences 5 (1738): 175–92. Bernoulli’s idea of utility arises from considering a
merchant, Sempronius, choosing whether to transport a valuable cargo in one ship or
to split it
between two, assuming that each ship has a 50 percent probability of sinking on the
journey. The
expected monetary value of the two solutions is the same, but Sempronius clearly prefers
the
two-ship solution.
(16)
By most accounts, von Neumann did not himself invent this architecture but his name
was on an early draft of an influential report describing the EDVAC storedprogram
computer.
(17)
The work of von Neumann and Morgenstern is in many ways the foundation of modern
economic theory: John von Neumann and Oskar Morgenstern, Theory of Games
and Economic Behavior (Princeton University Press, 1944).
(18)
The proposal that utility is a sum of discounted rewards was put forward as a
mathematically convenient hypothesis by Paul Samuelson, “A note on measurement of
utility,”
Review of Economic Studies 4 (1937): 155–61. If s0, s1, … is a sequence of states, then its utility in this
model is U(s0, s1, …) = ΣtγtR(st), where
γ is a discount factor and R is a reward function
describing the desirability of a state. Naïve application of this model seldom agrees
with
the judgment of real individuals about the desirability of present and future rewards.
For a
thorough analysis, see Shane Frederick, George Loewenstein, and Ted O’Donoghue, “Time
discounting
and time preference: A critical review,” Journal of Economic
Literature 40 (2002): 351–401.
(19)
Maurice Allais, a French economist, proposed a decision scenario in which humans
appear consistently to violate the von Neumann-Morgenstern axioms: Maurice Allais,
“Le
comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes
de l’école
américaine,” Econometrica 21 (1953):
503–46.
(20)
For an introduction to non-quantitative decision analysis, see Michael Wellman,
“Fundamental concepts of qualitative probabilistic networks,” Artificial
Intelligence 44 (1990): 257–303.
(21)
I will discuss the evidence for human irrationality further in Chapter 9. The
standard references include the following: Allais, “Le comportement”; Daniel Ellsberg,
Risk, Ambiguity, and Decision (PhD thesis, Harvard University, 1962);
Amos Tversky and Daniel Kahneman, “Judgment under uncertainty: Heuristics and biases,”
Science 185 (1974): 1124–31.
(22)
It should be clear that this is a thought experiment that cannot be realized in
practice. Choices about different futures are never presented in full detail, and
humans never
have the luxury of minutely examining and savoring those futures before choosing.
Instead, one is
given only brief summaries, such as “librarian” or “coal miner.” In making such a
choice, one is
really being asked to compare two probability distributions over complete futures,
one beginning
with the choice “librarian” and the other “coal miner,” with each distribution assuming
optimal
actions on one’s own part within each future. Needless to say, this is not
easy.
(23)
The first mention of a randomized strategy for games appears in Pierre Rémond de
Montmort, Essay d’analyse sur les jeux de hazard, 2nd ed. (Chez
Jacques Quillau, 1713). The book identifies a certain Monsieur de Waldegrave as the
source of an
optimal randomized solution for the card game Le Her. Details of Waldegrave’s identity
are
revealed by David Bellhouse, “The problem of Waldegrave,” Electronic
Journal for History of Probability and Statistics 3 (2007).
(24)
The problem is fully defined by specifying the probability that Alice scores in each
of four cases: when she shoots to Bob’s right and he dives right or left, and when
she shoots to
his left and he dives right or left. In this case, these probabilities are 25 percent,
70 percent,
65 percent, and 10 percent respectively. Now suppose that Alice’s strategy is to shoot
to Bob’s
right with probability p and his left with probability 1 −
p, while Bob dives to his right with probability q and left with probability 1 − q.
The payoff to Alice is UA =
0.25pq + 0.70 p(1 −
q) + 0.65 (1 − p)q + 0.10(1 − p) (1 − q), while Bob’s payoff is UB = −UA. At equilibrium, ∂UA/∂p = 0 and
∂UB/∂q = 0, giving p = 0.55 and
q = 0.60.
(25)
The original game-theoretic problem was introduced by Merrill Flood and Melvin
Dresher at the RAND Corporation; Tucker saw the payoff matrix on a visit to their
offices and
proposed a “story” to go along with it.
(26)
Game theorists typically say that Alice and Bob could cooperate with each other (refuse to talk) or defect and rat on their accomplice. I find this language confusing, because
“cooperate with each other” is not a choice that each agent can make separately, and
because in
common parlance one often talks about cooperating with the police, receiving a lighter
sentence in
return for cooperating, and so on.
(27)
For an interesting trust-based solution to the prisoner’s dilemma and other games,
see Joshua Letchford, Vincent Conitzer, and Kamal Jain, “An ‘ethical’ game-theoretic
solution
concept for two-player perfect-information games,” in Proceedings of the
4th International Workshop on Web and Internet Economics, ed. Christos
Papadimitriou and Shuzhong Zhang (Springer, 2008).
(28)
Origin of the tragedy of the commons: William Forster Lloyd, Two Lectures on the Checks to Population (Oxford University,
1833).
(29)
Modern revival of the topic in the context of global ecology: Garrett Hardin, “The
tragedy of the commons,” Science 162 (1968):
1243–48.
(30)
It’s quite possible that even if we had tried to build intelligent machines from chemical
reactions or biological cells, those assemblages would have turned out to be implementations
of Turing
machines in nontraditional materials. Whether an object is a generalpurpose computer
has nothing to do
with what it’s made of.
(31)
Turing’s breakthrough paper defined what is now known as the Turing machine, the basis for modern computer science. The Entscheidungsproblem, or decision problem, in the
title is the problem of deciding entailment in first-order logic: Alan Turing, “On
computable numbers,
with an application to the Entscheidungsproblem,” Proceedings of the London Mathematical Society, 2nd ser., 42 (1936):
230–65.
(32)
A good survey of research on negative capacitance by one of its inventors: Sayeef
Salahuddin, “Review of negative capacitance transistors,” in International
Symposium on VLSI Technology, Systems and Application (IEEE Press,
2016).
(33)
For a much better explanation of quantum computation, see Scott Aaronson, Quantum Computing since Democritus (Cambridge University Press,
2013).
(34)
The paper that established a clear complexity-theoretic distinction between classical
and
quantum computation: Ethan Bernstein and Umesh Vazirani, “Quantum complexity theory,”
SIAM Journal on Computing 26 (1997): 1411–73.
(35)
The following article by a renowned physicist provides a good introduction to the
current
state of understanding and technology: John Preskill, “Quantum computing in the NISQ
era and beyond,”
arXiv:1801.00862 (2018).
(36)
On the maximum computational ability of a one-kilogram object: Seth Lloyd, “Ultimate
physical limits to computation,” Nature 406 (2000):
1047–54.
(37)
For an example of the suggestion that humans may be the pinnacle of physically
achievable intelligence, see Kevin Kelly, “The myth of a superhuman AI,” Wired, April 25, 2017: “We tend to believe that the limit is way beyond us, way
‘above’ us, as we are ‘above’ an ant … What evidence do we have that the limit is
not
us?”
(38)
In case you are wondering about a simple trick to solve the halting problem: the
obvious method of just running the program to see if it finishes doesn’t work, because
that method
doesn’t necessarily finish. You might wait a million years and still not know if the
program is
really stuck in an infinite loop or just taking its time.
(39)
The proof that the halting problem is undecidable is an elegant piece of trickery.
The question: Is there a LoopChecker
(40)
I say “appear” because, as yet, the claim that the class of NP-complete problems
requires superpolynomial time (usually referred to as P ≠ NP) is still an unproven
conjecture.
After almost fifty years of research, however, nearly all mathematicians and computer
scientists
are convinced the claim is true.
(41)
Lovelace’s writings on computation appear mainly in her notes attached to her translation
of an Italian engineer’s commentary on Babbage’s engine: L. F. Menabrea, “Sketch of
the Analytical
Engine invented by Charles Babbage,” trans. Ada, Countess of Lovelace, in Scientific Memoirs, vol. III, ed. R. Taylor (R. and J. E. Taylor, 1843). Menabrea’s
original article, written in French and based on lectures given by Babbage in 1840,
appears in
Bibliothèque Universelle de Genève 82
(1842).
(42)
One of the seminal early papers on the possibility of artificial intelligence: Alan
Turing, “Computing machinery and intelligence,” Mind 59 (1950):
433–60.
(43)
The Shakey project at SRI is summarized in a retrospective by one of its leaders:
Nils
Nilsson, “Shakey the robot,” technical note 323 (SRI International, 1984). A twentyfour-minute
film,
SHAKEY: Experimentation in Robot Learning and Planning, was
made in 1969 and garnered national attention.
(44)
The book that marked the beginning of modern, probability-based AI: Judea Pearl,
Probabilistic Reasoning in Intelligent Systems: Networks of Plausible
Inference (Morgan Kaufmann, 1988).
(45)
Technically, chess is not fully observable. A program does need to remember a small
amount of information to determine the legality of castling and en passant moves and
to define draws
by repetition or by the fifty-move rule.
(46)
For a complete exposition, see Chapter 2 of Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd ed. (Pearson,
2010).
(47)
The size of the state space for StarCraft is discussed by Santiago Ontañon et al.,
“A
survey of real-time strategy game AI research and competition in StarCraft,” IEEE Transactions on Computational Intelligence and AI in Games 5 (2013): 293–311.
Vast numbers of moves are possible because a player can move all units simultaneously.
The numbers go
down as restrictions are imposed on how many units or groups of units can be moved
at
once.
(48)
On human-machine competition in StarCraft: Tom Simonite, “DeepMind beats pros at
StarCraft in another triumph for bots,” Wired, January 25,
2019.
(49)
AlphaZero is described by David Silver et al., “Mastering chess and shogi by self-play
with a general reinforcement learning algorithm,” arXiv:1712.01815 (2017).
(50)
Optimal paths in graphs are found using the A* algorithm and
its many descendants: Peter Hart, Nils Nilsson, and Bertram Raphael, “A formal basis
for the heuristic
determination of minimum cost paths,” IEEE Transactions on Systems Science and
Cybernetics SSC-4 (1968): 100–107.
(51)
The paper that introduced the Advice Taker program and logic-based knowledge systems:
John McCarthy, “Programs with common sense,” in Proceedings of the Symposium
on Mechanisation of Thought Processes (Her Majesty’s Stationery Office,
1958).
(52)
To get some sense of the significance of knowledge-based systems, consider database
systems. A database contains concrete, individual facts, such as the location of my
keys and the
identities of your Facebook friends. Database systems cannot store general rules,
such as the rules of
chess or the legal definition of British citizenship. They can count how many people
called Alice have
friends called Bob, but they cannot determine whether a particular Alice meets the
conditions for
British citizenship or whether a particular sequence of moves on a chessboard will
lead to checkmate.
Database systems cannot combine two pieces of knowledge to produce a third: they support
memory but
not reasoning. (It is true that many modern database systems provide a way to add
rules and a way to
use those rules to derive new facts; to the extent that they do, they are really knowledge-based
systems.) Despite being highly constricted versions of knowledge-based systems, database
systems
underlie most of present-day commercial activity and generate hundreds of billions
of dollars in value
every year.
(53)
The original paper describing the completeness theorem for first-order logic: Kurt
Gödel,
“Die Vollständigkeit der Axiome des logischen Funktionenkalküls,” Monatshefte
für Mathematik 37 (1930): 349–60.
(54)
The reasoning algorithm for first-order logic does have a gap: if there is no answer
—
that is, if the available knowledge is insufficient to give an answer either way —
then the algorithm
may never finish. This is unavoidable: it is mathematically impossible for a correct algorithm always to
terminate with “don’t know,” for essentially the same reason that no algorithm can
solve the halting
problem (page 37).
(55)
The first algorithm for theorem-proving in first-order logic worked by reducing
firstorder sentences to (very large numbers of) propositional sentences: Martin Davis
and Hilary
Putnam, “A computing procedure for quantification theory,” Journal of the
ACM 7 (1960): 201–15. Robinson’s resolution algorithm operated directly on first-order
logical sentences, using “unification” to match complex expressions containing logical
variables: J.
Alan Robinson, “A machine-oriented logic based on the resolution principle,” Journal of the ACM 12 (1965): 23–41.
(56)
One might wonder how Shakey the logical robot ever reached any definite conclusions
about
what to do. The answer is simple: Shakey’s knowledge base contained false assertions.
For example,
Shakey believed that by executing “push object A through door D into room B,” object
A would end up in
room B. This belief was false because Shakey could get stuck in the doorway or miss
the doorway
altogether or someone might sneakily remove object A from Shakey’s grasp. Shakey’s
plan execution
module could detect plan failure and replan accordingly, so Shakey was not, strictly
speaking, a
purely logical system.
(57)
An early commentary on the role of probability in human thinking: Pierre-Simon Laplace,
Essai philosophique sur les probabilités (Mme. Ve. Courcier,
1814).
(58)
Bayesian logic described in a fairly nontechnical way: Stuart Russell, “Unifying logic
and probability,” Communications of the ACM 58 (2015): 88–97. The
paper draws heavily on the PhD thesis research of my former student Brian
Milch.
(59)
The original source for Bayes’ theorem: Thomas Bayes and Richard Price, “An essay
towards
solving a problem in the doctrine of chances,” Philosophical Transactions of
the Royal Society of London 53 (1763): 370–418.
(60)
Technically, Samuel’s program did not treat winning and losing as absolute rewards;
by
fixing the value of material to be positive; however, the program generally tended
to work towards
winning.
(61)
The application of reinforcement learning to produce a world-class backgammon program:
Gerald Tesauro, “Temporal difference learning and TD-Gammon,” Communications
of the ACM 38 (1995): 58–68.
(62)
The DQN system that learns to play a wide variety of video games using deep RL: Volodymyr
Mnih et al., “Human-level control through deep reinforcement learning,” Nature 518 (2015): 529–33.
(63)
Bill Gates’s remarks on Dota 2 AI: Catherine Clifford, “Bill Gates says gamer bots
from
Elon Musk-backed nonprofit are ‘huge milestone’ in A.I.,” CNBC, June 28, 2018.
(64)
An account of OpenAI Five’s victory over the human world champions at Dota 2: Kelsey
Piper, “AI triumphs against the world’s top pro team in strategy game Dota 2,” Vox, April 13, 2019.
(65)
A compendium of cases in the literature where misspecification of reward functions
led to
unexpected behavior: Victoria Krakovna, “Specification gaming examples in AI,” Deep Safety (blog), April 2, 2018.
(66)
A case where an evolutionary fitness function defined in terms of maximum velocity
led to
very unexpected results: Karl Sims, “Evolving virtual creatures,” in Proceedings of the 21st Annual Conference on Computer Graphics and Interactive
Techniques (ACM, 1994).
(67)
For a fascinating exposition of the possibilities of reflex agents, see Valentino
Braitenberg, Vehicles: Experiments in Synthetic Psychology (MIT
Press, 1984).
(68)
News article on a fatal accident involving a vehicle in autonomous mode that hit a
pedestrian: Devin Coldewey, “Uber in fatal crash detected pedestrian but had emergency
braking
disabled,” TechCrunch, May 24, 2018.
(69)
On steering control algorithms, see, for example, Jarrod Snider, “Automatic steering
methods for autonomous automobile path tracking,” technical report CMU-RI-TR-09-08,
Robotics
Institute, Carnegie Mellon University, 2009.
(70)
Norfolk and Norwich terriers are two categories in the ImageNet database. They are
notoriously hard to tell apart and were viewed as a single breed until 1964.
(71)
A very unfortunate incident with image labeling: Daniel Howley, “Google Photos mislabels
2 black Americans as gorillas,” Yahoo Tech, June 29,
2015.
(72)
Follow-up article on Google and gorillas: Tom Simonite, “When it comes to gorillas,
Google Photos remains blind,” Wired, January 11,
2018.
الفصل الثالث: كيف قد يتطوَّر الذكاء الاصطناعي في المُستقبل؟
(1)
The basic plan for game-playing algorithms was laid out by Claude Shannon, “Programming
a
computer for playing chess,” Philosophical Magazine, 7th ser., 41
(1950): 256–75.
(2)
See figure 5٫12 of Stuart Russell and Peter Norvig, Artificial
Intelligence: A Modern Approach, 1st ed. (Prentice Hall, 1995). Note that the rating of
chess players and chess programs is not an exact science. Kasparov’s highest-ever
Elo rating was 2851,
achieved in 1999, but current chess engines such as Stockfish are rated at 3300 or
more.
(3)
The earliest reported autonomous vehicle on a public road: Ernst Dickmanns and Alfred
Zapp, “Autonomous high speed road vehicle guidance by computer vision,” IFAC Proceedings Volumes 20 (1987): 221–26.
(4)
The safety record for Google (subsequently Waymo) vehicles: “Waymo safety report:
On
the road to fully self-driving,” 2018.
(5)
So far there have been at least two driver fatalities and one pedestrian fatality.
Some references follow, along with brief quotes describing what happened. Danny Yadron
and Dan
Tynan, “Tesla driver dies in first fatal crash while using autopilot mode,” Guardian, June 30, 2016: “The autopilot sensors on the Model S failed
to distinguish a white tractor-trailer crossing the highway against a bright sky.”
Megan Rose
Dickey, “Tesla Model X sped up in Autopilot mode seconds before fatal crash, according
to NTSB,”
TechCrunch, June 7, 2018: “At 3 seconds prior to the crash
and up to the time of impact with the crash attenuator, the Tesla’s speed increased
from 62 to
70٫8 mph, with no precrash braking or evasive steering movement detected.” Devin Coldewey,
“Uber
in fatal crash detected pedestrian but had emergency braking disabled,” TechCrunch, May 24, 2018: “Emergency braking maneuvers are not enabled while the
vehicle is under computer control, to reduce the potential for erratic vehicle
behavior.”
(6)
The Society of Automotive Engineers (SAE) defines six levels of automation, where
Level 0 is none at all and Level 5 is full automation: “The full-time performance
by an automatic
driving system of all aspects of the dynamic driving task under all roadway and environmental
conditions that can be managed by a human driver.”
(7)
Forecast of economic effects of automation on transportation costs: Adele Peters,
“It
could be 10 times cheaper to take electric robo-taxis than to own a car by 2030,”
Fast Company, May 30, 2017.
(8)
The impact of accidents on the prospects for regulatory action on autonomous
vehicles: Richard Waters, “Self-driving car death poses dilemma for regulators,” Financial Times, March 20, 2018.
(9)
The impact of accidents on public perception of autonomous vehicles: Cox Automotive,
“Autonomous vehicle awareness rising, acceptance declining, according to Cox Automotive
mobility
study,” August 16, 2018.
(10)
The original chatbot: Joseph Weizenbaum, “ELIZA — a computer program for the study
of
natural language communication between man and machine,” Communications of
the ACM 9 (1966): 36–45.
(11)
See physiome.org for current activities in physiological modeling. Work in the 1960s
assembled models with thousands of differential equations: Arthur Guyton, Thomas Coleman,
and
Harris Granger, “Circulation: Overall regulation,” Annual Review of
Physiology 34 (1972): 13–44.
(12)
Some of the earliest work on tutoring systems was done by Pat Suppes and colleagues
at Stanford: Patrick Suppes and Mona Morningstar, “Computer-assisted instruction,”
Science 166 (1969): 343–50.
(13)
Michael Yudelson, Kenneth Koedinger, and Geoffrey Gordon, “Individualized Bayesian
knowledge tracing models,” in Artificial Intelligence in Education: 16th
International Conference, ed. H. Chad Lane et al. (Springer,
2013).
(14)
For an example of machine learning on encrypted data, see, for example, Reza Shokri
and Vitaly Shmatikov, “Privacy-preserving deep learning,” in Proceedings
of the 22nd ACM SIGSAC Conference on Computer and Communications Security (ACM,
2015).
(15)
A retrospective on the first smart home, based on a lecture by its inventor, James
Sutherland: James E. Tomayko, “Electronic Computer for Home Operation (ECHO): The
first home
computer,” IEEE Annals of the History of Computing 16 (1994):
59–61.
(16)
Summary of a smart-home project based on machine learning and automated decisions:
Diane Cook et al., “MavHome: An agent-based smart home,” in Proceedings of
the 1st IEEE International Conference on Pervasive Computing and Communications
(IEEE, 2003).
(17)
For the beginnings of an analysis of user experiences in smart homes, see Scott
Davidoff et al., “Principles of smart home control,” in Ubicomp 2006:
Ubiquitous Computing, ed. Paul Dourish and Adrian Friday (Springer,
2006).
(18)
Commercial announcement of AI-based smart homes: “The Wolff Company unveils
revolutionary smart home technology at new Annadel Apartments in Santa Rosa, California,”
Business Insider, March 12, 2018.
(19)
Article on robot chefs as commercial products: Eustacia Huen, “The world’s first home
robotic chef can cook over 100 meals,” Forbes, October 31,
2016.
(20)
Report from my Berkeley colleagues on deep RL for robotic motor control: Sergey
Levine et al., “End-to-end training of deep visuomotor policies,” Journal
of Machine Learning Research 17 (2016): 1–40.
(21)
On the possibilities for automating the work of hundreds of thousands of warehouse
workers: Tom Simonite, “Grasping robots compete to rule Amazon’s warehouses,” Wired, July 26, 2017.
(22)
I’m assuming a generous one laptop-CPU minute per page, or about
1011 operations. A third-generation tensor processing unit from
Google runs at about 1017 operations per second, meaning that it can
read a million pages per second, or about five hours for eighty million two-hundred-page
books.
(23)
A 2003 study on the global volume of information production by all channels: Peter
Lyman and Hal Varian, “How much information?”
sims.berkeley.edu/research/projects/how-much-info-2003.
(24)
For details on the use of speech recognition by intelligence agencies, see Dan
Froomkin, “How the NSA converts spoken words into searchable text,” The
Intercept, May 5, 2015.
(25)
Analysis of visual imagery from satellites is an enormous task: Mike Kim, “Mapping
poverty from space with the World Bank,” Medium.com, January 4, 2017. Kim estimates
eight million
people working 24/7, which converts to more than thirty million people working forty
hours per
week. I suspect this is an overestimate in practice, because the vast majority of
the images would
exhibit negligible change over the course of one day. On the other hand, the US intelligence
community employs tens of thousands of people sitting in vast rooms staring at satellite
images
just to keep track of what’s happening in small regions of interest; so one million
people is
probably about right for the whole world.
(26)
There is substantial progress towards a global observatory based on real-time
satellite image data: David Jensen and Jillian Campbell, “Digital earth: Building,
financing and
governing a digital ecosystem for planetary data,” white paper for the UN Science-Policy-Business
Forum on the Environment, 2018.
(27)
Luke Muehlhauser has written extensively on AI predictions, and I am indebted to him
for
tracking down original sources for the quotations that follow. See Luke Muehlhauser,
“What should we
learn from past AI forecasts?” Open Philanthropy Project report,
2016.
(28)
A forecast of the arrival of human-level AI within twenty years: Herbert Simon, The New Science of Management Decision (Harper & Row,
1960).
(29)
A forecast of the arrival of human-level AI within a generation: Marvin Minsky, Computation: Finite and Infinite Machines (Prentice Hall,
1967).
(30)
John McCarthy’s forecast of the arrival of human-level AI within “five to 500 years”:
Ian
Shenker, “Brainy robots in our future, experts think,” Detroit Free
Press, September 30, 1977.
(31)
For a summary of surveys of AI researchers on their estimates for the arrival of
humanlevel AI, see aiimpacts.org. An extended discussion of survey results on human-level
AI is given
by Katja Grace et al., “When will AI exceed human performance? Evidence from AI experts,”
arXiv:1705.08807v3 (2018).
(32)
For a chart mapping raw computer power against brain power, see Ray Kurzweil, “The
law of
accelerating returns,” Kurzweilai.net, March 7, 2001.
(33)
The Allen Institute’s Project Aristo: allenai.org/aristo.
(34)
For an analysis of the knowledge required to perform well on fourth-grade tests of
comprehension and common sense, see Peter Clark et al., “Automatic construction of
inference-supporting knowledge bases,” in Proceedings of the Workshop on
Automated Knowledge Base Construction (2014), akbc.ws/2014.
(35)
The NELL project on machine reading is described by Tom Mitchell et al., “Neverending
learning,” Communications of the ACM 61 (2018):
103–15.
(36)
The idea of bootstrapping inferences from text is due to Sergey Brin, “Extracting
patterns and relations from the World Wide Web,” in The World Wide Web and
Databases, ed. Paolo Atzeni, Alberto Mendelzon, and Giansalvatore Mecca (Springer,
1998).
(37)
For a visualization of the black-hole collision detected by LIGO, see LIGO Lab
Caltech, “Warped space and time around colliding black holes,” February 11, 2016,
youtube.com/watch?v=1agm33iEAuo.
(38)
The first publication describing observation of gravitational waves: Abbott et al.,
“Observation of gravitational waves from a binary black hole Physical
Review Letters 116 (2016): 061102.
(39)
On babies as scientists: Alison Gopnik, Andrew Meltzoff, Patricia Kuhl, The Scientist in the Crib: Minds, Brains, and How Children Learn
(William Morrow, 1999).
(40)
A summary of several projects on automated scientific analysis of experimental data
to discover laws: Patrick Langley et al., Scientific Discovery:
Computational Explorations of the Creative Processes (MIT Press,
1987).
(41)
Some early work on machine learning guided by prior knowledge: Stuart Russell,
The Use of Knowledge in Analogy and Induction (Pitman,
1989).
(42)
Goodman’s philosophical analysis of induction remains a source of inspiration: Nelson
Goodman, Fact, Fiction, and Forecast (University of London
Press, 1954).
(43)
A veteran AI researcher complains about mysticism in the philosophy of science:
Herbert Simon, “Explaining the ineffable: AI on the topics of intuition, insight and
inspiration,”
in Proceedings of the 14th International Conference on Artificial
Intelligence, ed. Chris Mellish (Morgan Kaufmann, 1995).
(44)
A survey of inductive logic programming by two originators of the field: Stephen
Muggleton and Luc de Raedt, “Inductive logic programming: Theory and methods,” Journal of Logic Programming 19-20 (1994):
629–79.
(45)
For an early mention of the importance of encapsulating complex operations as new
primitive actions, see Alfred North Whitehead, An Introduction to
Mathematics (Henry Holt, 1911).
(46)
Work demonstrating that a simulated robot can learn entirely by itself to stand up:
John Schulman et al., “High-dimensional continuous control using generalized advantage
estimation,” arXiv:1506.02438 (2015). A video demonstration is available at
youtube.com/watch?v=SHLuf2ZBQSw.
(47)
A description of a reinforcement learning system that learns to play a
capture-the-flag video game: Max Jaderberg et al., “Human-level performance in first-person
multiplayer games with population-based deep reinforcement learning,” arXiv:1807.01281
(2018).
(48)
A view of AI progress over the next few years: Peter Stone et al., “Artificial
intelligence and life in 2030,” One Hundred Year Study on Artificial
Intelligence, report of the 2015 Study Panel, 2016.
(49)
The media-fueled argument between Elon Musk and Mark Zuckerberg: Peter Holley,
“Billionaire burn: Musk says Zuckerberg’s understanding of AI threat ‘is limited,’”
The Washington Post, July 25, 2017.
(50)
On the value of search engines to individual users: Erik Brynjolfsson, Felix Eggers,
and
Avinash Gannamaneni, “Using massive online choice experiments to measure changes in
well-being,”
working paper no. 24514, National Bureau of Economic Research, 2018.
(51)
Penicillin was discovered several times and its curative powers were described in
medical
publications, but no one seems to have noticed. See
en.wikipedia.org/wiki/History_of_penicillin.
(52)
For a discussion of some of the more esoteric risks from omniscient, clairvoyant AI
systems, see David Auerbach, “The most terrifying thought experiment of all time,”
Slate, July 17, 2014.
(53)
An analysis of some potential pitfalls in thinking about advanced AI: Kevin Kelly,
“The
myth of a superhuman AI,” Wired, April 25,
2017.
(54)
Machines may share some aspects of cognitive structure
with humans, particularly those aspects dealing with perception and manipulation of
the physical world
and the conceptual structures involved in natural language understanding. Their deliberative
processes
are likely to be quite different because of the enormous disparities in
hardware.
(55)
According to 2016 survey data, the eighty-eighth percentile corresponds to
$100,000 per year: American Community Survey, US Census Bureau,
www.census.gov/programs-surveys/acs. For the same year, global per capita GDP was
$10,133: National Accounts Main Aggregates Database, UN Statistics Division,
unstats.un.org/unsd/snaama.
(56)
If the GDP growth phases in over ten years or twenty years, it’s worth $9,400
trillion or $6,800 trillion, respectively — still nothing to sneeze at. On an interesting
historical note, I. J. Good, who popularized the notion of an intelligence explosion
(page 142),
estimated the value of human-level AI to be at least “one megaKeynes,” referring to
the fabled
economist John Maynard Keynes. The value of Keynes’s contributions was estimated in
1963 as £100
billion, so a megaKeynes comes out to around $2,200,000 trillion in 2016 dollars.
Good pinned
the value of AI primarily on its potential to ensure that the human race survives
indefinitely. Later,
he came to wonder whether he should have added a minus sign.
(57)
The EU announced plans for $24 billion in research and development spending for
the period 2019–20. See European Commission, “Artificial intelligence: Commission
outlines a European
approach to boost investment and set ethical guidelines,” press release, April 25,
2018. China’s
long-term investment plan for AI, announced in 2017, envisages a core AI industry
generating
$150 billion annually by 2030. See, for example, Paul Mozur, “Beijing wants A.I. to
be made in
China by 2030,” The New York Times, July 20,
2017.
(58)
See, for example, Rio Tinto’s Mine of the Future program at
riotinto.com/australia/pilbara/mine-of-the-future-9603.aspx.
(59)
A retrospective analysis of economic growth: Jan Luiten van Zanden et al., eds.,
How Was Life? Global Well-Being since 1820 (OECD Publishing,
2014).
(60)
The desire for relative advantage over others, rather than an absolute quality of
life,
is a positional good; see Chapter 9.
الفصل الرابع: إساءة استخدام الذكاء الاصطناعي
(1)
Wikipedia’s article on the Stasi has several useful references on its workforce and
its overall impact on East German life.
(2)
For details on Stasi files, see Cullen Murphy, God’s Jury:
The Inquisition and the Making of the Modern World (Houghton Mifflin Harcourt,
2012).
(3)
For a thorough analysis of AI surveillance systems, see Jay Stanley, The Dawn of Robot Surveillance (American Civil Liberties Union,
2019).
(4)
Recent books on surveillance and control include Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New
Frontier of Power (PublicAffairs, 2019) and Roger McNamee, Zucked: Waking Up to the Facebook Catastrophe (Penguin Press,
2019).
(5)
News article on a blackmail bot: Avivah Litan, “Meet Delilah — the first insider
threat Trojan,” Gartner Blog Network, July 14, 2016.
(6)
For a low-tech version of human susceptibility to
misinformation, in which an unsuspecting individual becomes
convinced that the world is being destroyed by meteor strikes, see
Derren Brown: Apocalypse,
“Part One,” directed by Simon Dinsell, 2012,
youtube.com/watch?v=o_CUrMJOxqs.
(7)
An economic analysis of reputation systems and their corruption is given by Steven
Tadelis, “Reputation and feedback systems in online platform markets,” Annual Review of Economics 8 (2016): 321–40.
(8)
Goodhart’s law: “Any observed statistical regularity will tend to collapse once
pressure is placed upon it for control purposes.” For example, there may once have
been a
correlation between faculty quality and faculty salary, so the US News
& World Report college rankings measure faculty quality by faculty salaries.
This has contributed to a salary arms race that benefits faculty members but not the
students who
pay for those salaries. The arms race changes faculty salaries in a way that does
not depend on
faculty quality, so the correlation tends to disappear.
(9)
An article describing German efforts to police public discourse: Bernhard Rohleder,
“Germany set out to delete hate speech online. Instead, it made things worse,” World-Post, February 20, 2018.
(10)
On the “infopocalypse”: Aviv Ovadya, “What’s worse than fake news? The distortion
of
reality itself,” WorldPost, February 22,
2018.
(11)
On the corruption of online hotel reviews: Dina Mayzlin, Yaniv Dover, and Judith
Chevalier, “Promotional reviews: An empirical investigation review manipulation,”
American Economic Review 104 (2014): 2421–55.
(12)
Statement of Germany at the Meeting of the Group of Governmental Experts, Convention
on
Certain Conventional Weapons, Geneva, April 10, 2018.
(13)
The Slaughterbots movie, funded by the Future of Life
Institute, appeared in November 2017 and is available at
youtube.com/watch?v=9CO6M2HsoIA.
(14)
For a report on one of the bigger faux pas in military
public relations, see Dan Lamothe, “Pentagon agency wants drones to hunt in packs,
like wolves,”
The Washington Post, January 23, 2015.
(15)
Announcement of a large-scale drone swarm experiment: US Department of Defense,
“Department of Defense announces successful micro-drone demonstration,” news release
no. NR-008-17,
January 9, 2017.
(16)
Examples of research centers studying the impact of technology on employment are the
Work
and Intelligent Tools and Systems group at Berkeley, the Future of Work and Workers
project at the
Center for Advanced Study in the Behavioral Sciences at Stanford, and the Future of
Work Initiative at
Carnegie Mellon University.
(17)
A pessimistic take on future technological unemployment: Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future (Basic
Books, 2015).
(18)
Calum Chace, The Economic Singularity: Artificial Intelligence
and the Death of Capitalism (Three Cs, 2016).
(19)
For an excellent collection of essays, see Ajay Agrawal, Joshua Gans, and Avi Goldfarb,
eds., The Economics of Artificial Intelligence: An Agenda (National
Bureau of Economic Research, 2019).
(20)
The mathematical analysis behind this “inverted-U” employment curve is given by James
Bessen, “Artificial intelligence and jobs: The role of demand” in The
Economics of Artificial Intelligence, ed. Agrawal, Gans, and
Goldfarb.
(21)
For a discussion of economic dislocation arising from automation, see Eduardo Porter,
“Tech is splitting the US work force in two,” The New York Times,
February 4, 2019. The article cites the following report for this conclusion: David
Autor and Anna
Salomons, “Is automation labor-displacing? Productivity growth, employment, and the
labor share,”
Brookings Papers on Economic Activity
(2018).
(22)
For data on the growth of banking in the twentieth century, see Thomas Philippon,
“The
evolution of the US financial industry from 1860 to 2007: Theory and evidence,” working
paper,
2008.
(23)
The bible for jobs data and the growth and decline of occupations: US Bureau of Labor
Statistics, Occupational Outlook Handbook: 2018–2019 Edition
(Bernan Press, 2018).
(24)
A report on trucking automation: Lora Kolodny, “Amazon is hauling cargo in selfdriving
trucks developed by Embark,” CNBC, January 30, 2019.
(25)
The progress of automation in legal analytics, describing the results of a contest:
Jason
Tashea, “AI software is more accurate, faster than attorneys when assessing NDAs,”
ABA Journal, February 26, 2018.
(26)
A commentary by a distinguished economist, with a title explicitly evoking Keynes’s
1930
article: Lawrence Summers, “Economic possibilities for our children,” NBER
Reporter (2013).
(27)
The analogy between data science employment and a small lifeboat for a giant cruise
ship
comes from a discussion with Yong Ying-I, head of Singapore’s Public Service Division.
She conceded
that it was correct on the global scale, but noted that “Singapore is small enough
to fit in the
lifeboat.”
(28)
Support for UBI from a conservative viewpoint: Sam Bowman, “The ideal welfare system
is a
basic income,” Adam Smith Institute, November 25, 2013.
(29)
Support for UBI from a progressive viewpoint: Jonathan Bartley, “The Greens endorse
a
universal basic income. Others need to follow,” The Guardian, June
2, 2017.
(30)
Chace, in The Economic Singularity, calls the
“paradise” version of UBI the Star Trek economy, noting that in the
more recent series of Star Trek episodes, money has been abolished
because technology has created essentially unlimited material goods and energy. He
also points to the
massive changes in economic and social organization that will be needed to make such
a system
successful.
(31)
The economist Richard Baldwin also predicts a future of personal services in his book
The Globotics Upheaval: Globalization, Robotics, and the Future of
Work (Oxford University Press, 2019).
(32)
The book that is viewed as having exposed the failure of “whole-word” literacy education
and launched decades of struggle between the two main schools of thought on reading:
Rudolf Flesch,
Why Johnny Can’t Read: And What You Can Do about It (Harper
& Bros., 1955).
(33)
On educational methods that enable the recipient to adapt to the rapid rate of
technological and economic change in the next few decades: Joseph Aoun, Robot-Proof: Higher Education in the Age of Artificial Intelligence (MIT Press,
2017).
(34)
A radio lecture in which Turing predicted that humans would be overtaken by machines:
Alan Turing, “Can digital machines think?,” May 15, 1951, radio broadcast, BBC Third
Programme.
Typescript available at turingarchive.org.
(35)
News article describing the “naturalization” of Sophia as a citizen of Saudi Arabia:
Dave
Gershgorn, “Inside the mechanical brain of the world’s first robot citizen,” Quartz, November 12, 2017.
(36)
On Yann LeCun’s view of Sophia: Shona Ghosh, “Facebook’s AI boss described Sophia
the
robot as ‘complete b——t’ and ‘Wizard-of-Oz AI,’” Business Insider,
January 6, 2018.
(37)
An EU proposal on legal rights for robots: Committee on Legal Affairs of the European
Parliament, “Report with recommendations to the Commission on Civil Law Rules on Robotics
(2015/2103(INL)),” 2017.
(38)
The GDPR provision on a “right to an explanation” is not, in fact, new: it is very
similar to Article 15(1) of the 1995 Data Protection Directive, which it
supersedes.
(39)
Here are three recent papers providing insightful mathematical analyses of fairness:
Moritz Hardt, Eric Price, and Nati Srebro, “Equality of opportunity in supervised
learning,” in
Advances in Neural Information Processing Systems 29, ed.
Daniel Lee et al. (2016); Matt Kusner et al., “Counterfactual fairness,” in Advances in Neural Information Processing Systems 30, ed. Isabelle Guyon et al.
(2017); Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, “Inherent trade-offs
in the fair
determination of risk scores,” in 8th Innovations in Theoretical Computer
Science Conference, ed. Christos Papadimitriou (Dagstuhl Publishing,
2017).
(40)
News article describing the consequences of software failure for air traffic control:
Simon Calder, “Thousands stranded by flight cancellations after systems failure at
Europe’s
air-traffic coordinator,” The Independent, April 3,
2018.
الفصل الخامس: الذكاء الاصطناعي الفائق الذكاء
(1)
Lovelace wrote, “The Analytical Engine has no pretensions whatever to originate anything.
It can do whatever we know how to order it to perform. It can follow analysis; but
it has no power of
anticipating any analytical relations or truths.” This was one of the arguments against
AI that was
refuted by Alan Turing, “Computing machinery and intelligence,” Mind 59 (1950): 433–60.
(2)
The earliest known article on existential risk from AI was by Richard Thornton, “The
age
of machinery,” Primitive Expounder IV (1847):
281.
(3)
“The Book of the Machines” was based on an earlier
article by Samuel Butler, “Darwin among the machines,” The Press
(Christchurch, New Zealand), June 13, 1863.
(4)
Another lecture in which Turing predicted the subjugation of humankind: Alan Turing,
“Intelligent machinery, a heretical theory” (lecture given to the 51 Society, Manchester,
1951).
Typescript available at turingarchive.org.
(5)
Wiener’s prescient discussion of technological control over humanity and a plea to
retain
human autonomy: Norbert Wiener, The Human Use of Human Beings
(Riverside Press, 1950).
(6)
The front-cover blurb from Wiener’s 1950 book is remarkably similar to the motto of
the
Future of Life Institute, an organization dedicated to studying the existential risks
that humanity
faces: “Technology is giving life the potential to flourish like never before … or
to
self-destruct.”
(7)
An updating of Wiener’s views arising from his increased appreciation of the possibility
of intelligent machines: Norbert Wiener, God and Golem, Inc.: A Comment on
Certain Points Where Cybernetics Impinges on Religion (MIT Press,
1964).
(8)
Asimov’s Three Laws of Robotics first appeared in Isaac Asimov, “Runaround,” Astounding Science Fiction, March 1942. The laws are as
follows:
(1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.
(2) A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
(3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
It is important to understand that Asimov proposed these laws as a way to generate interesting story plots, not as a serious guide for future roboticists. Several of his stories, including “Runaround,” illustrate the problematic consequences of taking the laws literally. From the standpoint of modern AI, the laws fail to acknowledge any element of probability and risk: the legality of robot actions that expose a human to some probability of harm — however infinitesimal — is therefore unclear.
(9)
The notion of instrumental goals is due to Stephen Omohundro, “The nature of
selfimproving artificial intelligence” (unpublished manuscript, 2008). See also Stephen
Omohundro,
“The basic AI drives,” in Artificial General Intelligence 2008: Proceedings of
the First AGI Conference, ed. Pei Wang, Ben Goertzel, and Stan Franklin (IOS Press,
2008).
(10)
The objective of Johnny Depp’s character, Will Caster, seems to be to solve the problem
of physical reincarnation so that he can be reunited with his wife, Evelyn. This just
goes to show
that the nature of the overarching objective doesn’t matter — the instrumental goals
are all the
same.
(11)
The original source for the idea of an intelligence explosion: I. J. Good, “Speculations
concerning the first ultraintelligent machine,” in Advances in
Computers, vol. 6, ed. Franz Alt and Morris Rubinoff (Academic Press,
1965).
(12)
An example of the impact of the intelligence explosion idea: Luke Muehlhauser, in
Facing the Intelligence Explosion (intelligenceexplosion.com),
writes, “Good’s paragraph ran over me like a train.”
(13)
Diminishing returns can be illustrated as follows: suppose that a 16 percent improvement
in intelligence creates a machine capable of making an 8 percent improvement, which
in turn creates a
4 percent improvement, and so on. This process reaches a limit at about 36 percent
above the original
level. For more discussion on these issues, see Eliezer Yudkowsky, “Intelligence explosion
microeconomics,” technical report 2013-1, Machine Intelligence Research Institute,
2013.
(14)
For a view of AI in which humans become irrelevant, see Hans Moravec, Mind Children: The Future of Robot and Human Intelligence (Harvard
University Press, 1988). See also Hans Moravec, Robot: Mere Machine to
Transcendent Mind (Oxford University Press, 2000).
الفصل السادس: الجدل غير الواسع الدائر حول الذكاء الاصطناعي
(1)
A serious publication provides a serious review of Bostrom’s Superintelligence: Paths, Dangers, Strategies: “Clever cogs,” Economist, August 9, 2014.
(2)
A discussion of myths and misunderstandings concerning the risks of AI: Scott Alexander,
“AI researchers on AI risk,” Slate Star Codex (blog), May 22,
2015.
(3)
The classic work on multiple dimensions of intelligence: Howard Gardner, Frames of Mind: The Theory of Multiple Intelligences (Basic Books,
1983).
(4)
On the implications of multiple dimensions of intelligence for the possibility of
superhuman AI: Kevin Kelly, “The myth of a superhuman AI,” Wired, April 25, 2017.
(5)
Evidence that chimpanzees have better short-term memory than humans: Sana Inoue and
Tetsuro Matsuzawa, “Working memory of numerals in chimpanzees,” Current
Biology 17 (2007), R1004–5.
(6)
An important early work questioning the prospects for rule-based AI systems: Hubert
Dreyfus, What Computers Can’t Do (MIT Press,
1972).
(7)
The first in a series of books seeking physical explanations for consciousness and
raising doubts about the ability of AI systems to achieve real intelligence: Roger
Penrose,
The Emperor’s New Mind: Concerning Computers, Minds, and the Laws of
Physics (Oxford University Press, 1989).
(8)
A revival of the critique of AI based on the incompleteness theorem: Luciano Floridi,
“Should we be afraid of AI?” Aeon, May 9,
2016.
(9)
A revival of the critique of AI based on the Chinese room argument: John Searle,
“What your computer can’t know,” The New York Review of Books,
October 9, 2014.
(10)
A report from distinguished AI researchers claiming that superhuman AI is probably
impossible: Peter Stone et al., “Artificial intelligence and life in 2030,” One Hundred
Year Study
on Artificial Intelligence, report of the 2015 Study Panel, 2016.
(11)
News article based on Andrew Ng’s dismissal of risks from AI: Chris Williams, “AI
guru Ng: Fearing a rise of killer robots is like worrying about overpopulation on
Mars,” Register, March 19, 2015.
(12)
An example of the “experts know best” argument: Oren Etzioni, “It’s time to
intelligently discuss artificial intelligence,” Backchannel,
December 9, 2014.
(13)
News article claiming that real AI researchers dismiss talk of risks: Erik Sofge,
“Bill Gates fears AI, but AI researchers know better,” Popular
Science, January 30, 2015.
(14)
Another claim that real AI researchers dismiss AI risks: David Kenny, “IBM’s open
letter to Congress on artificial intelligence,” June 27, 2017,
ibm.com/blogs/policy/kenny-artificial-intelligence-letter.
(15)
Report from the workshop that proposed voluntary restrictions on genetic engineering:
Paul Berg et al., “Summary statement of the Asilomar Conference on Recombinant DNA
Molecules,”
Proceedings of the National Academy of Sciences 72 (1975):
1981–84.
(16)
Policy statement arising from the invention of CRISPR-Cas9 for gene editing:
Organizing Committee for the International Summit on Human Gene Editing, “On human
gene editing:
International Summit statement,” December 3, 2015.
(17)
The latest policy statement from leading biologists: Eric Lander et al., “Adopt a
moratorium on heritable genome editing,” Nature 567 (2019):
165–68.
(18)
Etzioni’s comment that one cannot mention risks if one does not also mention benefits
appears alongside his analysis of survey data from AI researchers: Oren Etzioni, “No,
the experts
don’t think superintelligent AI is a threat to humanity,” MIT Technology
Review, September 20, 2016. In his analysis he argues that anyone who expects
superhuman AI to take more than twenty-five years — which includes this author as
well as Nick
Bostrom — is not concerned about the risks of AI.
(19)
A news article with quotations from the Musk-Zuckerberg “debate”: Alanna Petroff,
“Elon Musk says Mark Zuckerberg’s understanding of AI is ‘limited,’” CNN
Money, July 25, 2017.
(20)
In 2015 the Information Technology and Innovation Foundation organized a debate
titled “Are super intelligent computers really a threat to humanity?” Robert Atkinson,
director of
the foundation, suggests that mentioning risks is likely to result in reduced funding
for AI.
Video available at
itif.org/events/2015/06/30/are-super-intelligent-computers-really-threat-humanity;
the relevant
discussion begins at 41:30.
(21)
A claim that our culture of safety will solve the AI control problem without ever
mentioning it: Steven Pinker, “Tech prophecy and the underappreciated causal power
of ideas,” in
Possible Minds: Twenty-Five Ways of Looking at AI, ed. John
Brockman (Penguin Press, 2019).
(22)
For an interesting analysis of Oracle AI, see Stuart Armstrong, Anders Sandberg, and
Nick Bostrom, “Thinking inside the box: Controlling and using an Oracle AI,” Minds and Machines 22 (2012): 299–324.
(23)
Views on why AI is not going to take away jobs: Kenny, “IBM’s open
letter.”
(24)
An example of Kurzweil’s positive views of merging human brains with AI: Ray
Kurzweil, interview by Bob Pisani, June 5, 2015, Exponential Finance Summit, New York,
NY.
(25)
Article quoting Elon Musk on neural lace: Tim Urban, “Neuralink and the brain’s
magical future,” Wait But Why, April 20, 2017.
(26)
For the most recent developments in Berkeley’s neural dust project, see David Piech
et al., “StimDust: A 1٫7 mm3, implantable wireless precision neural
stimulator with ultrasonic power and communication,” arXiv: 1807.07590
(2018).
(27)
Susan Schneider, in Artificial You: AI and the Future of Your
Mind (Princeton University Press, 2019), points out the risks of ignorance in
proposed technologies such as uploading and neural prostheses: that, absent any real
understanding
of whether electronic devices can be conscious and given the continuing philosophical
confusion
over persistent personal identity, we may inadvertently end our own conscious existences
or
inflict suffering on conscious machines without realizing that they are
conscious.
(28)
An interview with Yann LeCun on AI risks: Guia Marie Del Prado, “Here’s what
Facebook’s artificial intelligence expert thinks about the future,” Business Insider, September 23, 2015.
(29)
A diagnosis of AI control problems arising from an excess of testosterone: Steven
Pinker, “Thinking does not imply subjugating,” in What to Think About
Machines That Think, ed. John Brockman (Harper Perennial,
2015).
(30)
A seminal work on many philosophical topics, including the question of whether moral
obligations may be perceived in the natural world: David Hume, A Treatise
of Human Nature (John Noon, 1738).
(31)
An argument that a sufficiently intelligent machine cannot help but pursue human
objectives: Rodney Brooks, “The seven deadly sins of AI predictions,” MIT
Technology Review, October 6, 2017.
(32)
Pinker, “Thinking does not imply subjugating.”
(33)
For an optimistic view arguing that AI safety problems will necessarily be resolved
in our favor: Steven Pinker, “Tech prophecy.”
(34)
On the unsuspected alignment between “skeptics” and “believers” in AI risk: Alexander,
“AI researchers on AI risk.”
الفصل السابع: الذكاء الاصطناعي: توجُّه مُختلف
(1)
For a guide to detailed brain modeling, now slightly outdated, see Anders Sandberg
and Nick
Bostrom, “Whole brain emulation: A roadmap,” technical report 2008-3,
Future of Humanity Institute, Oxford University, 2008.
(2)
For an introduction to genetic programming from a leading exponent, see John Koza,
Genetic Programming: On the Programming of Computers by Means of Natural
Selection (MIT Press, 1992).
(3)
The parallel to Asimov’s Three Laws of Robotics is entirely
coincidental.
(4)
The same point is made by Eliezer Yudkowsky, “Coherent extrapolated volition,” technical
report, Singularity Institute, 2004. Yudkowsky argues that directly building in “Four
Great Moral
Principles That Are All We Need to Program into AIs” is a sure road to ruin for humanity.
His notion
of the “coherent extrapolated volition of humankind” has the same general flavor as
the first
principle; the idea is that a superintelligent AI system could work out what humans,
collectively,
really want.
(5)
You can certainly have preferences over whether a machine is helping you achieve your
preferences or you are achieving them through your own efforts. For example, suppose
you prefer
outcome A to outcome B, all other things being equal. You are unable to achieve outcome
A unaided, and
yet you still prefer B to getting A with the machine’s help. In that case the machine
should decide
not to help you — unless perhaps it can do so in a way that is completely undetectable
by you. You
may, of course, have preferences about undetectable help as well as detectable
help.
(6)
The phrase “the greatest good of the greatest number” originates in the work of
Francis Hutcheson, An Inquiry into the Original of Our Ideas of Beauty and
Virtue, In Two Treatises (D. Midwinter et al., 1725). Some have ascribed the
formulation to an earlier comment by Wilhelm Leibniz; see Joachim Hruschka, “The greatest
happiness principle and other early German anticipations of utilitarian theory,” Utilitas 3 (1991): 165–77.
(7)
One might propose that the machine should include terms for animals as well as humans
in its own objective function. If these terms have weights that correspond to how
much people care
about animals, then the end result will be the same as if the machine cares about
animals only
through caring about humans who care about animals. Giving each living animal equal
weight in the
machine’s objective function would certainly be catastrophic — for example, we are
outnumbered
fifty thousand to one by Antarctic krill and a billion trillion to one by
bacteria.
(8)
The moral philosopher Toby Ord made the same point to me in his comments on an early
draft of this book: “Interestingly, the same is true in the study of moral philosophy.
Uncertainty
about moral value of outcomes was almost completely neglected in moral philosophy
until very
recently. Despite the fact that it is our uncertainty of moral matters that leads
people to ask
others for moral advice and, indeed, to do research on moral philosophy at
all!”
(9)
One excuse for not paying attention to uncertainty about preferences is that it is
formally equivalent to ordinary uncertainty, in the following sense: being uncertain
about what I
like is the same as being certain that I like likable things while being uncertain
about what
things are likable. This is just a trick that appears to move the uncertainty into
the world, by
making “likability by me” a property of objects rather than a property of me. In game
theory, this
trick has been thoroughly institutionalized since the 1960s, following a series of
papers by my
late colleague and Nobel laureate John Harsanyi: “Games with incomplete information
played by
‘Bayesian’ players, Parts I–III,” Management Science 14 (1967,
1968): 159–82, 320–34, 486–502. In decision theory, the standard reference is the
following:
Richard Cyert and Morris de Groot, “Adaptive utility,” in Expected Utility
Hypotheses and the Allais Paradox, ed. Maurice Allais and Ole Hagen (D. Reidel,
1979).
(10)
AI researchers working in the area of preference elicitation are an obvious
exception. See, for example, Craig Boutilier, “On the foundations of expected expected utility,” in Proceedings of the 18th
International Joint Conference on Artificial Intelligence (Morgan Kaufmann, 2003).
Also Alan Fern et al., “A decision-theoretic model of assistance,” Journal
of Artificial Intelligence Research 50 (2014): 71–104.
(11)
A critique of beneficial AI based on a misinterpretation of a journalist’s brief
interview with the author in a magazine article: Adam Elkus, “How to be good: Why
you can’t teach
human values to artificial intelligence,” Slate, April 20,
2016.
(12)
The origin of trolley problems: Frank Sharp, “A study of the influence of custom on
the moral judgment,” Bulletin of the University of Wisconsin
236 (1908).
(13)
The “anti-natalist” movement believes it is morally wrong for humans to reproduce
because to live is to suffer and because humans’ impact on the Earth is profoundly
negative. If
you consider the existence of humanity to be a moral dilemma, then I suppose I do
want machines to
resolve this moral dilemma the right way.
(14)
Statement on China’s AI policy by Fu Ying, vice chair of the Foreign Affairs Committee
of
the National People’s Congress. In a letter to the 2018 World AI Conference in Shanghai,
Chinese
president Xi Jinping wrote, “Deepened international cooperation is required to cope
with new issues in
fields including law, security, employment, ethics and governance.” I am indebted
to Brian Tse for
bringing these statements to my attention.
(15)
A very interesting paper on the non-naturalistic non-fallacy, showing how preferences
can
be inferred from the state of the world as arranged by humans: Rohin Shah et al.,
“The implicit
preference information in an initial state,” in Proceedings of the 7th
International Conference on Learning Representations (2019),
iclr.cc/Conferences/2019/Schedule.
(16)
Retrospective on Asilomar: Paul Berg, “Asilomar 1975: DNA modification secured,”
Nature 455 (2008): 290–91.
(17)
News article reporting Putin’s speech on AI: “Putin: Leader in artificial intelligence
will rule world,” Associated Press, September 4, 2017.
الفصل الثامن: الذكاء الاصطناعي النافع على نحو مثبت
(1)
Fermat’s Last Theorem asserts that the equation
(2)
A paper from the Machine Intelligence Research Institute poses many related issues:
Scott
Garrabrant and Abram Demski, “Embedded agency,” AI Alignment Forum, November 15,
2018.
(3)
The classic work on multiattribute utility theory: Ralph Keeney and Howard Raiffa,
Decisions with Multiple Objectives: Preferences and Value
Tradeoffs (Wiley, 1976).
(4)
Paper introducing the idea of inverse RL: Stuart Russell, “Learning agents for uncertain
environments,” in Proceedings of the 11th Annual Conference on Computational
Learning Theory (ACM, 1998).
(5)
The original paper on structural estimation of Markov decision processes: Thomas Sargent,
“Estimation of dynamic labor demand schedules under rational expectations,” Journal of Political Economy 86 (1978): 1009–44.
(6)
The first algorithms for IRL: Andrew Ng and Stuart Russell, “Algorithms for inverse
reinforcement learning,” in Proceedings of the 17th International Conference
on Machine Learning, ed. Pat Langley (Morgan Kaufmann, 2000).
(7)
Better algorithms for inverse RL: Pieter Abbeel and Andrew Ng, “Apprenticeship learning
via inverse reinforcement learning,” in Proceedings of the 21st International
Conference on Machine Learning, ed. Russ Greiner and Dale Schuurmans (ACM Press,
2004).
(8)
Understanding inverse RL as Bayesian updating: Deepak Ramachandran and Eyal Amir,
“Bayesian inverse reinforcement learning,” in Proceedings of the 20th
International Joint Conference on Artificial Intelligence, ed. Manuela Veloso (AAAI
Press, 2007)
(9)
How to teach helicopters to fly and do aerobatic maneuvers: Adam Coates, Pieter Abbeel,
and Andrew Ng, “Apprenticeship learning for helicopter control,” Communications of the ACM 52 (2009): 97–105.
(10)
The original name proposed for an assistance game was a cooperative inverse reinforcement learning game, or CIRL game. See Dylan
Hadfield-Menell et al., “Cooperative inverse reinforcement learning,” in Advances in Neural Information Processing Systems 29, ed. Daniel Lee et al.
(2016).
(11)
These numbers are chosen just to make the game interesting.
(12)
The equilibrium solution to the game can be found by a process called iterated best response: pick any strategy for Harriet; pick the best
strategy for Robbie, given Harriet’s strategy; pick the best strategy for Harriet,
given Robbie’s
strategy; and so on. If this process reaches a fixed point, where neither strategy
changes, then
we have found a solution. The process unfolds as follows:
(1) Start with the greedy strategy for Harriet: make 2 paperclips if she prefers paperclips; make 1 of each if she is indifferent; make 2 staples if she prefers staples.
(2) There are three possibilities Robbie has to consider, given this strategy for Harriet:
(a) If Robbie sees Harriet make 2 paperclips, he infers that she prefers paperclips, so he now believes the value of a paperclip is uniformly distributed between 50 ȼ And $1٫00, with an average of 75 ȼ … In that case, his best plan is to make 90 paperclips with an expected value of $67٫50 for Harriet.
(b) If Robbie sees Harriet make 1 of each, he infers that she values paperclips and staples at 50 ȼ, so the best choice is to make 50 of each.
(c) If Robbie sees Harriet make 2 staples, then by the same argument as in 2(a), he should make 90 staples.
(3) Given this strategy for Robbie, Harriet’s best strategy is now somewhat different from the greedy strategy in step 1: if Robbie is going to respond to her making 1 of each by making 50 of each, then she is better off making 1 of each not just if she is exactly indifferent but if she is anywhere close to indifferent. In fact, the optimal policy is now to make 1 of each if she values paperclips anywhere between about 44٫6ȼ and 55٫4ȼ.
(4) Given this new strategy for Harriet, Robbie’s strategy remains unchanged. For example, if she chooses 1 of each, he infers that the value of a paperclip is uniformly distributed between 44٫6ȼ and 55٫4ȼ, with an average of 50ȼ, so the best choice is to make 50 of each. Because Robbie’s strategy is the same as in step 2, Harriet’s best response will be the same as in step 3, and we have found the equilibrium.
(13)
For a more complete analysis of the off-switch game, see Dylan Hadfield-Menell et
al., “The off-switch game,” in Proceedings of the 26th International Joint
Conference on Artificial Intelligence, ed. Carles Sierra (IJCAI,
2017).
(14)
The proof of the general result is quite simple if you don’t mind integral signs.
Let
P(u) be Robbie’s prior
probability density over Harriet’s utility for the proposed action a. Then the value of going ahead with a
is
(15)
Perhaps the next elaboration in line, for the one human–one robot case, is to
consider a Harriet who does not yet know her own preferences regarding some aspect
of the world,
or whose preferences have not yet been formed.
(16)
To see how exactly Robbie converges to an incorrect belief, consider a model in which
Harriet is slightly irrational, making errors with a probability that diminishes exponentially
as
the size of error increases. Robbie offers Harriet 4 paperclips in return for 1 staple;
she
refuses. According to Robbie’s beliefs, this is irrational: even at
25
ȼ
Per
paperclip and 75
ȼ
per staple, she should accept 4 for 1. Therefore, she
must have made a mistake — but this mistake is much more likely
if her true value is 25
ȼ
than if it is, say, 30
ȼ, because the error costs her a lot
more if her value for paperclips is 30ȼ … Now Robbie’s probability distribution has
25ȼ as
the most likely value because it represents the smallest error on Harriet’s part,
with
exponentially lower probabilities for values higher than 25ȼ … If he keeps trying the same
experiment, the probability distribution becomes more and more concentrated close
to
25ȼ …
In the limit, Robbie becomes certain that Harriet’s value for paperclips is
25ȼ.
(17)
Robbie could, for example, have a normal (Gaussian) distribution for his prior belief
about the exchange rate, which stretches from −∞ to +∞.
(18)
For an example of the kind of mathematical analysis that may be needed, see Avrim
Blum, Lisa Hellerstein, and Nick Littlestone, “Learning in the presence of finitely
or infinitely
many irrelevant attributes,” Journal of Computer and System
Sciences 50 (1995): 32–40. Also Lori Dalton, “Optimal Bayesian feature selection,”
in Proceedings of the 2013 IEEE Global Conference on Signal and
Information Processing, ed. Charles Bouman, Robert Nowak, and Anna Scaglione (IEEE,
2013).
(19)
Here I am rephrasing slightly a question by Moshe Vardi at the Asilomar Conference
on
Beneficial AI, 2017.
(20)
Michael Wellman and Jon Doyle, “Preferential semantics for goals,” in Proceedings of the 9th National Conference on Artificial Intelligence
(AAAI Press, 1991). This paper draws on a much earlier proposal by Georg von Wright,
“The logic of
preference reconsidered,” Theory and Decision 3 (1972):
140–67.
(21)
My late Berkeley colleague has the distinction of becoming an adjective. See Paul
Grice,
Studies in the Way of Words (Harvard University Press,
1989).
(22)
The original paper on direct stimulation of pleasure centers in the brain: James Olds
and
Peter Milner, “Positive reinforcement produced by electrical stimulation of septal
area and other
regions of rat brain,” Journal of Comparative and Physiological
Psychology 47 (1954): 419–27.
(23)
Letting rats push the button: James Olds, “Self-stimulation of the brain; its use
to
study local effects of hunger, sex, and drugs,” Science 127 (1958):
315–24.
(24)
Letting humans push the button: Robert Heath, “Electrical self-stimulation of the
brain
in man,” American Journal of Psychiatry 120 (1963):
571–77.
(25)
A first mathematical treatment of wireheading, showing how it occurs in reinforcement
learning agents: Mark Ring and Laurent Orseau, “Delusion, survival, and intelligent
agents,” in
Artificial General Intelligence: 4th International Conference,
ed. Jurgen Schmidhuber, Kristinn Thorisson, and Moshe Looks (Springer, 2011). One
possible solution to
the wireheading problem: Tom Everitt and Marcus Hutter, “Avoiding wireheading with
value reinforcement
learning,” arXiv:1605.03143 (2016).
(26)
How it might be possible for an intelligence explosion to occur safely: Benja Fallenstein
and Nate Soares, “Vingean reflection: Reliable reasoning for self-improving agents,”
technical report
2015-2, Machine Intelligence Research Institute, 2015.
(27)
The difficulty agents face in reasoning about themselves and their successors: Benja
Fallenstein and Nate Soares, “Problems of self-reference in self-improving space-time
embedded
intelligence,” in Artificial General Intelligence: 7th International
Conference, ed. Ben Goertzel, Laurent Orseau, and Javier Snaider (Springer,
2014).
(28)
Showing why an agent might pursue an objective different from its true objective if
its
computational abilities are limited: Jonathan Sorg, Satinder Singh, and Richard Lewis,
“Internal
rewards mitigate agent boundedness,” in Proceedings of the 27th International
Conference on Machine Learning, ed. Johannes Furnkranz and Thorsten Joachims (2010),
icml.cc/Conferences/2010/papers/icml2010proceedings.zip.
الفصل التاسع: التعقيدات: البشر
(1)
Some have argued that biology and neuroscience are also directly relevant. See, for
example,
Gopal Sarma, Adam Safron, and Nick Hay, “Integrative biological simulation, neuropsychology,
and AI
safety,” arxiv.org/abs/1811.03493 (2018).
(2)
On the possibility of making computers liable for damages: Paulius Čerka, Jurgita
Grigienė, and Gintarė Sirbikytė, “Liability for damages caused by artificial intelligence,”
Computer Law and Security Review 31 (2015):
376–89.
(3)
For an excellent machine-oriented introduction to standard ethical theories and their
implications for designing AI systems, see Wendell Wallach and Colin Allen, Moral Machines: Teaching Robots Right from Wrong (Oxford University
Press, 2008).
(4)
The sourcebook for utilitarian thought: Jeremy Bentham, An
Introduction to the Principles of Morals and Legislation (T. Payne & Son,
1789).
(5)
Mill’s elaboration of his tutor Bentham’s ideas was extraordinarily influential on
liberal thought: John Stuart Mill, Utilitarianism (Parker, Son
& Bourn, 1863).
(6)
The paper introducing preference utilitarianism and preference autonomy: John
Harsanyi, “Morality and the theory of rational behavior,” Social
Research 44 (1977): 623–56.
(7)
An argument for social aggregation via weighted sums of utilities when deciding on
behalf of multiple individuals: John Harsanyi, “Cardinal welfare, individualistic
ethics, and
interpersonal comparisons of utility,” Journal of Political
Economy 63 (1955): 309–21.
(8)
A generalization of Harsanyi’s social aggregation theorem to the case of unequal
prior beliefs: Andrew Critch, Nishant Desai, and Stuart Russell, “Negotiable reinforcement
learning for Pareto optimal sequential decision-making,” in Advances in
Neural Information Processing Systems 31, ed. Samy Bengio et al.
(2018).
(9)
The sourcebook for ideal utilitarianism: G. E. Moore, Ethics (Williams & Norgate, 1912).
(10)
News article citing Stuart Armstrong’s colorful example of misguided utility
maximization: Chris Matyszczyk, “Professor warns robots could keep us in coffins on
heroin drips,”
CNET, June 29, 2015.
(11)
Popper’s theory of negative utilitarianism (so named later by Smart): Karl Popper,
The Open Society and Its Enemies (Routledge,
1945).
(12)
A refutation of negative utilitarianism: R. Ninian Smart, “Negative utilitarianism,”
Mind 67 (1958): 542–43.
(13)
For a typical argument for risks arising from “end human suffering” commands, see
“Why do we think AI will destroy us?,” Reddit,
reddit.com/r/Futurology/comments/38fp6o/why_do_we_think_ai_will_destroy_us.
(14)
A good source for self-deluding incentives in AI: Ring and Orseau, “Delusion,
survival, and intelligent agents.”
(15)
On the impossibility of interpersonal comparisons of utility: W. Stanley Jevons,
The Theory of Political Economy (Macmillan,
1871).
(16)
The utility monster makes its appearance in Robert Nozick, Anarchy, State, and Utopia (Basic Books, 1974).
(17)
For example, we can fix immediate death to have a utility of 0 and a maximally happy
life to have a utility of 1. See John Isbell, “Absolute games,” in Contributions to the Theory of Games, vol. 4, ed. Albert Tucker and R. Duncan Luce
(Princeton University Press, 1959).
(18)
The oversimplified nature of Thanos’s population-halving policy is discussed by Tim
Harford, “Thanos shows us how not to be an economist,” Financial
Times, April 20, 2019. Even before the film debuted, defenders of Thanos began to
congregate on the subreddit r/thanosdidnothingwrong/. In keeping with the subreddit’s
motto,
350,000 of the 700,000 members were later purged.
(19)
On utilities for populations of different sizes: Henry Sidgwick, The Methods of Ethics (Macmillan, 1874).
(20)
The Repugnant Conclusion and other knotty problems of utilitarian thinking: Derek
Parfit, Reasons and Persons (Oxford University Press,
1984).
(21)
For a concise summary of axiomatic approaches to population ethics, see Peter
Eckersley, “Impossibility and uncertainty theorems in AI value alignment,” in Proceedings of the AAAI Workshop on Artificial Intelligence Safety,
ed. Huáscar Espinoza et al. (2019).
(22)
Calculating the long-term carrying capacity of the Earth: Daniel O’Neill et al., “A
good life for all within planetary boundaries,” Nature
Sustainability 1 (2018): 88–95.
(23)
For an application of moral uncertainty to population ethics, see Hilary Greaves and
Toby Ord, “Moral uncertainty about population axiology,” Journal of Ethics
and Social Philosophy 12 (2017): 135–67. A more comprehensive analysis is provided
by Will MacAskill, Krister Bykvist, and Toby Ord, Moral
Uncertainty (Oxford University Press, forthcoming).
(24)
Quotation showing that Smith was not so obsessed with selfishness as is commonly
imagined: Adam Smith, The Theory of Moral Sentiments (Andrew
Millar; Alexander Kincaid and J. Bell, 1759).
(25)
For an introduction to the economics of altruism, see Serge-Christophe Kolm and Jean
Ythier, eds., Handbook of the Economics of Giving, Altruism and
Reciprocity, 2 vols. (North-Holland, 2006).
(26)
On charity as selfish: James Andreoni, “Impure altruism and donations to public goods:
A
theory of warm-glow giving,” Economic Journal 100 (1990):
464–77.
(27)
For those who like equations: let Alice’s intrinsic well-being be measured by wA and Bob’s by wB. Then the utilities for Alice and Bob are defined as
follows:
(28)
Models in which each individual’s utility is a linear combination of everyone’s wellbeing
are just one possibility. Much more general models are possible — for example, models
in which some
individuals prefer to avoid severe inequalities in the distribution of well-being,
even at the expense
of reducing the total, while other individuals would really prefer that no one have
preferences about
inequality at all. Thus, the overall approach I am proposing accommodates multiple
moral theories held
by individuals; at the same time, it doesn’t insist that any one of those moral theories
is correct or
should have much sway over outcomes for those who hold a different theory. I am indebted
to Toby Ord
for pointing out this feature of the approach.
(29)
Arguments of this type have been made against policies designed to ensure equality
of
outcome, notably by the American legal philosopher Ronald Dworkin. See, for example,
Ronald Dworkin,
“What is equality? Part 1: Equality of welfare,” Philosophy and Public
Affairs 10 (1981): 185–246. I am indebted to Iason Gabriel for this
reference.
(30)
Malice in the form of revenge-based punishment for transgressions is certainly a common
tendency. Although it plays a social role in keeping members of a community in line,
it can be
replaced by an equally effective policy driven by deterrence and prevention — that
is, weighing the
intrinsic harm done when punishing the transgressor against the benefits to the larger
society.
(31)
Let EAB and PAB be Alice’s coefficients of envy and pride
respectively, and assume that they apply to the difference in well-being. Then a (somewhat
oversimplified) formula for Alice’s utility could be the following:
(32)
On the sociology of conspicuous consumption: Thorstein Veblen, The Theory of the Leisure Class: An Economic Study of Institutions (Macmillan,
1899).
(33)
Fred Hirsch, The Social Limits to Growth (Routledge
& Kegan Paul, 1977).
(34)
I am indebted to Ziyad Marar for pointing me to social identity theory and its importance
in understanding human motivation and behavior. See, for example, Dominic Abrams and
Michael Hogg,
eds., Social Identity Theory: Constructive and Critical Advances
(Springer, 1990). For a much briefer summary of the main ideas, see Ziyad Marar, “Social
identity,” in
This Idea Is Brilliant: Lost, Overlooked, and Underappreciated Scientific
Concepts Everyone Should Know, ed. John Brockman (Harper Perennial,
2018).
(35)
Here, I am not suggesting that we necessarily need a detailed understanding of the
neural
implementation of cognition; what is needed is a model at the “software” level of
how preferences,
both explicit and implicit, generate behavior. Such a model would need to incorporate
what is known
about the reward system.
(36)
Ralph Adolphs and David Anderson, The Neuroscience of Emotion: A
New Synthesis (Princeton University Press, 2018).
(37)
See, for example, Rosalind Picard, Affective
Computing, 2nd ed. (MIT Press, 1998).
(38)
Waxing lyrical on the delights of the durian: Alfred Russel Wallace, The Malay Archipelago: The Land of the Orang-Utan, and the Bird of
Paradise (Macmillan, 1869).
(39)
A less rosy view of the durian: Alan Davidson, The Oxford
Companion to Food (Oxford University Press, 1999). Buildings have been evacuated
and planes turned around in mid-flight because of the durian’s overpowering
odor.
(40)
I discovered after writing this chapter that the durian was used for exactly the same
philosophical purpose by Laurie Paul, Transformative Experience
(Oxford University Press, 2014). Paul suggests that uncertainty about one’s own preferences
presents fatal problems for decision theory, a view contradicted by Richard Pettigrew,
Transformative experience and decision theory, Philosophy and
Phenomenological Research 91 (2015): 766–74. Neither author refers to the early
work of Harsanyi, Games with incomplete information, Parts I–III, or Cyert and de
Groot,
Adaptive utility.
(41)
An initial paper on helping humans who don’t know their own preferences and are
learning about them: Lawrence Chan et al., “The assistive multi-armed bandit,” in
Proceedings of the 14th ACM/IEEE International Conference on Human–Robot
Interaction (HRI), ed. David Sirkin et al. (IEEE, 2019).
(42)
Eliezer Yudkowsky, in Coherent Extrapolated
Volition (Singularity Institute, 2004), lumps all these aspects, as well as plain
inconsistency, under the heading of muddle—a term that has not,
unfortunately, caught on.
(43)
On the two selves who evaluate experiences: Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus & Giroux,
2011).
(44)
Edgeworth’s hedonimeter, an imaginary device for measuring happiness moment to
moment: Francis Edgeworth, Mathematical Psychics: An Essay on the
Application of Mathematics to the Moral Sciences (Kegan Paul,
1881).
(45)
A standard text on sequential decisions under uncertainty: Martin Puterman, Markov Decision Processes: Discrete
Stochastic Dynamic Programming (Wiley, 1994).
(46)
On axiomatic assumptions that justify additive representations of utility over time:
Tjalling Koopmans, “Representation of preference orderings over time,” in Decision and Organization, ed. C. Bartlett McGuire, Roy Radner, and Kenneth Arrow
(North-Holland, 1972).
(47)
The 2019 humans (who might, in 2099, be long dead or might just be the earlier selves
of 2099 humans) might wish to build the machines in a way that respects the 2019 preferences
of
the 2019 humans rather than pandering to the undoubtedly shallow and ill-considered
preferences of
humans in 2099. This would be like drawing up a constitution that disallows any amendments.
If the
2099 humans, after suitable deliberation, decide they wish to override the preferences
built in by
the 2019 humans, it seems reasonable that they should be able to do so. After all,
it is they and
their descendants who have to live with the consequences.
(48)
I am indebted to Wendell Wallach for this observation.
(49)
An early paper dealing with changes in preferences over time: John Harsanyi, “Welfare
economics of variable tastes,” Review of Economic Studies 21
(1953): 204–13. A more recent (and somewhat technical) survey is provided by Franz
Dietrich and
Christian List, “Where do preferences come from?,” International Journal
of Game Theory 42 (2013): 613–37. See also Laurie Paul, Transformative Experience (Oxford University
Press, 2014), and Richard Pettigrew, “Choosing for Changing Selves,”
philpapers.org/archive/PETCFC.pdf.
(50)
For a rational analysis of irrationality, see Jon Elster, Ulysses and the Sirens: Studies in Rationality and Irrationality (Cambridge
University Press, 1979).
(51)
For promising ideas on cognitive prostheses for humans, see Falk Lieder, “Beyond
bounded rationality: Reverse-engineering and enhancing human intelligence” (PhD thesis,
University
of California, Berkeley, 2018).
الفصل العاشر: هل حُلَّت المشكلة؟
(1)
On the application of assistance games to driving: Dorsa Sadigh et al., “Planning
for
cars that coordinate with people,” Autonomous Robots 42 (2018):
1405–26.
(2)
Apple is, curiously, absent from this list. It does have an AI research group and
is
ramping up rapidly. Its traditional culture of secrecy means that its impact in the
marketplace of
ideas is quite limited so far.
(3)
Max Tegmark, interview, Do You Trust This Computer?,
directed by Chris Paine, written by Mark Monroe (2018).
(4)
On estimating the impact of cybercrime: “Cybercrime cost $600 billion and targets
banks first,” Security Magazine, February 21,
2018.
الملحق «أ»: البحث عن حلول
(1)
The basic plan for chess programs of the next sixty years: Claude Shannon, “Programming
a
computer for playing chess,” Philosophical Magazine, 7th ser., 41
(1950): 256–75. Shannon’s proposal drew on a centuries-long tradition of evaluating
chess positions by
adding up piece values; see, for example, Pietro Carrera, Il gioco degli
scacchi (Giovanni de Rossi, 1617).
(2)
A report describing Samuel’s heroic research on an early reinforcement learning algorithm
for checkers: Arthur Samuel, “Some studies in machine learning using the game of checkers,”
IBM Journal of Research and Development 3 (1959):
210–29.
(3)
The concept of rational metareasoning and its application to search and game playing
emerged from the thesis research of my student Eric Wefald, who died tragically in
a car accident
before he could write up his work; the following appeared posthumously: Stuart Russell
and Eric
Wefald, Do the Right Thing: Studies in Limited Rationality (MIT
Press, 1991). See also Eric Horvitz, “Rational metareasoning and compilation for optimizing
decisions
under bounded resources,” in Computational Intelligence, II: Proceedings of
the International Symposium, ed. Francesco Gardin and Giancarlo Mauri (North-Holland,
1990); and Stuart Russell and Eric Wefald, “On optimal game-tree search using rational
meta-reasoning,” in Proceedings of the 11th International Joint Conference on
Artificial Intelligence, ed. Natesa Sridharan (Morgan Kaufmann,
1989).
(4)
Perhaps the first paper showing how hierarchical organization reduces the combinatorial
complexity of planning: Herbert Simon, “The architecture of complexity,” Proceedings of the American Philosophical Society 106 (1962):
467–82.
(5)
The canonical reference for hierarchical planning is Earl Sacerdoti, “Planning in
a
hierarchy of abstraction spaces,” Artificial Intelligence 5 (1974):
115–35. See also Austin Tate, “Generating project networks,” in Proceedings of
the 5th International Joint Conference on Artificial Intelligence, ed. Raj Reddy
(Morgan Kaufmann, 1977).
(6)
A formal definition of what high-level actions do: Bhaskara Marthi, Stuart Russell,
and
Jason Wolfe, “Angelic semantics for high-level actions,” in Proceedings of the
17th International Conference on Automated Planning and Scheduling, ed. Mark Boddy,
Maria Fox, and Sylvie Thiebaux (AAAI Press, 2007).
الملحق «ب»: المعرفة والمنطق
(1)
This example is unlikely to be from Aristotle, but may have originated with Sextus
Empiricus,
who lived probably in the second or third century CE.
(2)
The first algorithm for theorem-proving in first-order logic worked by reducing
firstorder sentences to (very large numbers of) propositional sentences: Martin Davis
and Hilary
Putnam, “A computing procedure for quantification theory,” Journal of the
ACM 7 (1960): 201–15.
(3)
An improved algorithm for propositional inference: Martin Davis, George Logemann,
and
Donald Loveland, “A machine program for theorem-proving,” Communications of
the ACM 5 (1962): 394–97.
(4)
The satisfiability problem — deciding whether a collection of sentences is true in
some world — is NP-complete. The reasoning problem — deciding
whether a sentence follows from the known sentences — is co-NP-complete, a class that
is thought to be
harder than NP-complete problems.
(5)
There are two exceptions to this rule: no repetition (a stone may not be played that
returns the board to a situation that existed previously) and no suicide (a stone
may not be placed
such that it would immediately be captured — for example, if it is already
surrounded).
(6)
The work that introduced first-order logic as we understand it today (Begriffsschrift means “concept writing”): Gottlob Frege, Begriffsschrift, eine der arithmetischen nachgebildete Formelsprache des reinen
Denkens (Halle, 1879). Frege’s notation for first-order logic was so bizarre and
unwieldy that it was soon replaced by the notation introduced by Giuseppe Peano, which
remains in
common use today.
(7)
A summary of Japan’s bid for supremacy through knowledge-based systems: Edward Feigenbaum
and Pamela McCorduck, The Fifth Generation: Artificial Intelligence and
Japan’s Computer Challenge to the World (Addison-Wesley, 1983).
(8)
The US efforts included the Strategic Computing Initiative and the formation of the
Microelectronics and Computer Technology Corporation (MCC). See Alex Roland and Philip
Shiman,
Strategic Computing: DARPA and the Quest for Machine Intelligence,
1983–1993 (MIT Press, 2002).
(9)
A history of Britain’s response to the re-emergence of AI in the 1980s: Brian Oakley
and
Kenneth Owen, Alvey: Britain’s Strategic Computing Initiative (MIT
Press, 1990).
(10)
The origin of the term GOFAI: John Haugeland,
Artificial Intelligence: The Very Idea (MIT Press,
1985).
(11)
Interview with Demis Hassabis on the future of AI and deep learning: Nick Heath, “Google
DeepMind founder Demis Hassabis: Three truths about AI,” TechRepublic, September 24, 2018.
الملحق «ﺟ»: عدم اليقين والاحتمال
(1)
Pearl’s work was recognized by the Turing Award in 2011.
(2)
Bayes nets in more detail: Every node in the network is annotated with the probability
of
each possible value, given each possible combination of values for the node’s parents (that is, those nodes that point to it). For example, the probability that
Doubles12 has value true is 1٫0 when D1 and D2 have the same value, and 0٫0 otherwise. A possible world is an assignment of values
to all the nodes. The probability of such a world is the product of the
appropriate probabilities from each of the nodes.
(3)
A compendium of applications of Bayes nets: Olivier Pourret, Patrick Naïm, and Bruce
Marcot, eds., Bayesian Networks: A Practical Guide to Applications
(Wiley, 2008).
(4)
The basic paper on probabilistic programming: Daphne Koller, David McAllester, and
Avi
Pfeffer, “Effective Bayesian inference for stochastic programs,” in Proceedings of the 14th National Conference on Artificial Intelligence (AAAI Press,
1997). For many additional references, see probabilistic-programming.org.
(5)
Using probabilistic programs to model human concept learning: Brenden Lake, Ruslan
Salakhutdinov, and Joshua Tenenbaum, “Human-level concept learning through probabilistic
program
induction,” Science 350 (2015): 1332–38.
(6)
For a detailed description of the seismic monitoring application and associated
probability model, see Nimar Arora, Stuart Russell, and Erik Sudderth, “NET-VISA:
Network processing
vertically integrated seismic analysis,” Bulletin of the Seismological Society
of America 103 (2013): 709–29.
(7)
News article describing one of the first serious self-driving car crashes: Ryan Randazzo,
“Who was at fault in self-driving Uber crash? Accounts in Tempe police report disagree,”
Republic (azcentral.com), March 29, 2017.
الملحق «د»: التعلم من التجربة
(1)
The foundational discussion of inductive learning: David Hume, Philosophical Essays Concerning Human Understanding (A. Millar,
1748).
(2)
Leslie Valiant, “A theory of the learnable,” Communications of
the ACM 27 (1984): 1134–42. See also Vladimir Vapnik, Statistical Learning Theory (Wiley, 1998). Valiant’s approach concentrated on
computational complexity, Vapnik’s on statistical analysis of the learning capacity
of various classes
of hypotheses, but both shared a common theoretical core connecting data and predictive
accuracy.
(3)
For example, to learn the difference between the “situational superko” and “natural
situational superko” rules, the learning algorithm would have to try repeating a board
position that
it had created previously by a pass rather than by playing a stone. The results would
be different in
different countries.
(4)
For a description of the ImageNet competition, see Olga Russakovsky et al., “ImageNet
large scale visual recognition challenge,” International Journal of Computer
Vision 115 (2015): 211–52.
(5)
The first demonstration of deep networks for vision: Alex Krizhevsky, Ilya Sutskever,
and
Geoffrey Hinton, “ImageNet classification with deep convolutional neural networks,”
in Advances in Neural Information Processing Systems 25, ed. Fernando
Pereira et al. (2012).
(6)
The difficulty of distinguishing over one hundred breeds of dogs: Andrej Karpathy,
“What
I learned from competing against a ConvNet on ImageNet,” Andrej Karpathy
Blog, September 2, 2014.
(7)
Blog post on inceptionism research at Google: Alexander Mordvintsev, Christopher Olah,
and Mike Tyka, “Inceptionism: Going deeper into neural networks,” Google AI
Blog, June 17, 2015. The idea seems to have originated with J. P. Lewis, “Creation by
refinement: A creativity paradigm for gradient descent learning networks,” in Proceedings of the IEEE International Conference on Neural Networks (IEEE,
1988).
(8)
News article on Geoff Hinton having second thoughts about deep networks: Steve LeVine,
“Artificial intelligence pioneer says we need to start over,” Axios, September 15, 2017.
(9)
A catalog of shortcomings of deep learning: Gary Marcus, “Deep learning: A critical
appraisal,” arXiv:1801.00631 (2018).
(10)
A popular textbook on deep learning, with a frank assessment of its weaknesses: François
Chollet, Deep Learning with Python (Manning Publications,
2017).
(11)
An explanation of explanation-based learning: Thomas Dietterich, “Learning at the
knowledge level,” Machine Learning 1 (1986):
287–315.
(12)
A superficially quite different explanation of explanation-based learning: John Laird,
Paul Rosenbloom, and Allen Newell, “Chunking in Soar: The anatomy of a general learning
mechanism,”
Machine Learning 1 (1986): 11–46.