زائر

تسجيل الاشتراك

لديك حساب؟ تسجيل دخول

  • ePub Logo
  • PDF Logo
  • Kindle Logo

ملاحظات

الفصل الأول: ماذا لو نجحنا؟

(1)
The first edition of my textbook on AI, co-authored with Peter Norvig, currently director of research at Google: Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 1st ed. (Prentice Hall, 1995).
(2)
Robinson developed the resolution algorithm, which can, given enough time, prove any logical consequence of a set of first-order logical assertions. Unlike previous algorithms, it did not require conversion to propositional logic. J. Alan Robinson, “A machine-oriented logic based on the resolution principle,” Journal of the ACM 12 (1965): 23–41.
(3)
Arthur Samuel, an American pioneer of the computer era, did his early work at IBM. The paper describing his work on checkers was the first to use the term machine learning, although Alan Turing had already talked about “a machine that can learn from experience” as early as 1947. Arthur Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development 3 (1959): 210–29.
(4)
The “Lighthill Report,” as it became known, led to the termination of research funding for AI except at the universities of Edinburgh and Sussex: Michael James Lighthill, “Artificial intelligence: A general survey,” in Artificial Intelligence: A Paper Symposium (Science Research Council of Great Britain, 1973).
(5)
The CDC 6600 filled an entire room and cost the equivalent of $20 million. For its era it was incredibly powerful, albeit a million times less powerful than an iPhone.
(6)
Following Deep Blue’s victory over Kasparov, at least one commentator predicted that it would take one hundred years before the same thing happened in Go: George Johnson, “To test a powerful computer, play an ancient game,” The New York Times, July 29, 1997.
(7)
For a highly readable history of the development of nuclear technology, see Richard Rhodes, The Making of the Atomic Bomb (Simon & Schuster, 1987).
(8)
A simple supervised learning algorithm may not have this effect, unless it is wrapped within an A/B testing framework (as is common in online marketing settings). Bandit algorithms and reinforcement learning algorithms will have this effect if they operate with an explicit representation of user state or an implicit representation in terms of the history of interactions with the user.
(9)
Some have argued that profit-maximizing corporations are already out-of-control artificial entities. See, for example, Charles Stross, “Dude, you broke the future!” (keynote, 34th Chaos Communications Congress, 2017). See also Ted Chiang, “Silicon Valley is turning into its own worst fear,” Buzzfeed, December 18, 2017. The idea is explored further by Daniel Hillis, “The first machine intelligences,” in Possible Minds: Twenty-Five Ways of Looking at AI, ed. John Brockman (Penguin Press, 2019).
(10)
For its time, Wiener’s paper was a rare exception to the prevailing view that all technological progress was a good thing: Norbert Wiener, “Some moral and technical consequences of automation,” Science 131 (1960): 1355–58.

الفصل الثاني: مفهوم الذكاء في البشر والآلات

(1)
Santiago Ramon y Cajal proposed synaptic changes as the site of learning in 1894, but it was not until the late 1960s that this hypothesis was confirmed experimentally. See Timothy Bliss and Terje Lomo, “Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path,” Journal of Physiology 232 (1973): 331–56.
(2)
For a brief introduction, see James Gorman, “Learning how little we know about the brain,” The New York Times, November 10, 2014. See also Tom Siegfried, “There’s a long way to go in understanding the brain,” ScienceNews, July 25, 2017. A special 2017 issue of the journal Neuron (vol. 94, pp. 933–1040) provides a good overview of many different approaches to understanding the brain.
(3)
The presence or absence of consciousness — actual subjective experience — certainly makes a difference in our moral consideration for machines. If ever we gain enough understanding to design conscious machines or to detect that we have done so, we would face many important moral issues for which we are largely unprepared.
(4)
The following paper was among the first to make a clear connection between reinforcement learning algorithms and neurophysiological recordings: Wolfram Schultz, Peter Dayan, and P. Read Montague, “A neural substrate of prediction and reward,” Science 275 (1997): 1593–99.
(5)
Studies of intracranial stimulation were carried out with the hope of finding cures for various mental illnesses. See, for example, Robert Heath, “Electrical self-stimulation of the brain in man,” American Journal of Psychiatry 120 (1963): 571–77.
(6)
An example of a species that may be facing self-extinction via addiction: Bryson Voirin, “Biology and conservation of the pygmy sloth, Bradypus pygmaeus,” Journal of Mammalogy 96 (2015): 703–7.
(7)
The Baldwin effect in evolution is usually attributed to the following paper: James Baldwin, “A new factor in evolution,” American Naturalist 30 (1896): 441–51.
(8)
The core idea of the Baldwin effect also appears in the following work: Conwy Lloyd Morgan, Habit and Instinct (Edward Arnold, 1896).
(9)
A modern analysis and computer implementation demonstrating the Baldwin effect: Geoffrey Hinton and Steven Nowlan, “How learning can guide evolution,” Complex Systems 1 (1987): 495–502.
(10)
Further elucidation of the Baldwin effect by a computer model that includes the evolution of the internal reward-signaling circuitry: David Ackley and Michael Littman, “Interactions between learning and evolution,” in Artificial Life II, ed. Christopher Langton et al. (Addison-Wesley, 1991).
(11)
Here I am pointing to the roots of our present-day concept of intelligence, rather than describing the ancient Greek concept of nous, which had a variety of related meanings.
(12)
The quotation is taken from Aristotle, Nicomachean Ethics, Book III, 3, 1112b.
(13)
Cardano, one of the first European mathematicians to consider negative numbers, developed an early mathematical treatment of probability in games. He died in 1576, eighty-seven years before his work appeared in print: Gerolamo Cardano, Liber de ludo aleae (Lyons, 1663).
(14)
Arnauld’s work, initially published anonymously, is often called The Port-Royal Logic: Antoine Arnauld, La logique, ou l’art de penser (Chez Charles Savreux, 1662). See also Blaise Pascal, Pensées (Chez Guillaume Desprez, 1670).
(15)
The concept of utility: Daniel Bernoulli, “Specimen theoriae novae de mensura sortis,” Proceedings of the St. Petersburg Imperial Academy of Sciences 5 (1738): 175–92. Bernoulli’s idea of utility arises from considering a merchant, Sempronius, choosing whether to transport a valuable cargo in one ship or to split it between two, assuming that each ship has a 50 percent probability of sinking on the journey. The expected monetary value of the two solutions is the same, but Sempronius clearly prefers the two-ship solution.
(16)
By most accounts, von Neumann did not himself invent this architecture but his name was on an early draft of an influential report describing the EDVAC storedprogram computer.
(17)
The work of von Neumann and Morgenstern is in many ways the foundation of modern economic theory: John von Neumann and Oskar Morgenstern, Theory of Games and Economic Behavior (Princeton University Press, 1944).
(18)
The proposal that utility is a sum of discounted rewards was put forward as a mathematically convenient hypothesis by Paul Samuelson, “A note on measurement of utility,” Review of Economic Studies 4 (1937): 155–61. If s0, s1, … is a sequence of states, then its utility in this model is U(s0, s1, …) = ΣtγtR(st), where γ is a discount factor and R is a reward function describing the desirability of a state. Naïve application of this model seldom agrees with the judgment of real individuals about the desirability of present and future rewards. For a thorough analysis, see Shane Frederick, George Loewenstein, and Ted O’Donoghue, “Time discounting and time preference: A critical review,” Journal of Economic Literature 40 (2002): 351–401.
(19)
Maurice Allais, a French economist, proposed a decision scenario in which humans appear consistently to violate the von Neumann-Morgenstern axioms: Maurice Allais, “Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école américaine,” Econometrica 21 (1953): 503–46.
(20)
For an introduction to non-quantitative decision analysis, see Michael Wellman, “Fundamental concepts of qualitative probabilistic networks,” Artificial Intelligence 44 (1990): 257–303.
(21)
I will discuss the evidence for human irrationality further in Chapter 9. The standard references include the following: Allais, “Le comportement”; Daniel Ellsberg, Risk, Ambiguity, and Decision (PhD thesis, Harvard University, 1962); Amos Tversky and Daniel Kahneman, “Judgment under uncertainty: Heuristics and biases,” Science 185 (1974): 1124–31.
(22)
It should be clear that this is a thought experiment that cannot be realized in practice. Choices about different futures are never presented in full detail, and humans never have the luxury of minutely examining and savoring those futures before choosing. Instead, one is given only brief summaries, such as “librarian” or “coal miner.” In making such a choice, one is really being asked to compare two probability distributions over complete futures, one beginning with the choice “librarian” and the other “coal miner,” with each distribution assuming optimal actions on one’s own part within each future. Needless to say, this is not easy.
(23)
The first mention of a randomized strategy for games appears in Pierre Rémond de Montmort, Essay d’analyse sur les jeux de hazard, 2nd ed. (Chez Jacques Quillau, 1713). The book identifies a certain Monsieur de Waldegrave as the source of an optimal randomized solution for the card game Le Her. Details of Waldegrave’s identity are revealed by David Bellhouse, “The problem of Waldegrave,” Electronic Journal for History of Probability and Statistics 3 (2007).
(24)
The problem is fully defined by specifying the probability that Alice scores in each of four cases: when she shoots to Bob’s right and he dives right or left, and when she shoots to his left and he dives right or left. In this case, these probabilities are 25 percent, 70 percent, 65 percent, and 10 percent respectively. Now suppose that Alice’s strategy is to shoot to Bob’s right with probability p and his left with probability 1 − p, while Bob dives to his right with probability q and left with probability 1 − q. The payoff to Alice is UA = 0.25pq + 0.70 p(1 − q) + 0.65 (1 − p)q + 0.10(1 − p) (1 − q), while Bob’s payoff is UB = −UA. At equilibrium, ∂UA/∂p = 0 and ∂UB/∂q = 0, giving p = 0.55 and q = 0.60.
(25)
The original game-theoretic problem was introduced by Merrill Flood and Melvin Dresher at the RAND Corporation; Tucker saw the payoff matrix on a visit to their offices and proposed a “story” to go along with it.
(26)
Game theorists typically say that Alice and Bob could cooperate with each other (refuse to talk) or defect and rat on their accomplice. I find this language confusing, because “cooperate with each other” is not a choice that each agent can make separately, and because in common parlance one often talks about cooperating with the police, receiving a lighter sentence in return for cooperating, and so on.
(27)
For an interesting trust-based solution to the prisoner’s dilemma and other games, see Joshua Letchford, Vincent Conitzer, and Kamal Jain, “An ‘ethical’ game-theoretic solution concept for two-player perfect-information games,” in Proceedings of the 4th International Workshop on Web and Internet Economics, ed. Christos Papadimitriou and Shuzhong Zhang (Springer, 2008).
(28)
Origin of the tragedy of the commons: William Forster Lloyd, Two Lectures on the Checks to Population (Oxford University, 1833).
(29)
Modern revival of the topic in the context of global ecology: Garrett Hardin, “The tragedy of the commons,” Science 162 (1968): 1243–48.
(30)
It’s quite possible that even if we had tried to build intelligent machines from chemical reactions or biological cells, those assemblages would have turned out to be implementations of Turing machines in nontraditional materials. Whether an object is a generalpurpose computer has nothing to do with what it’s made of.
(31)
Turing’s breakthrough paper defined what is now known as the Turing machine, the basis for modern computer science. The Entscheidungsproblem, or decision problem, in the title is the problem of deciding entailment in first-order logic: Alan Turing, “On computable numbers, with an application to the Entscheidungsproblem,” Proceedings of the London Mathematical Society, 2nd ser., 42 (1936): 230–65.
(32)
A good survey of research on negative capacitance by one of its inventors: Sayeef Salahuddin, “Review of negative capacitance transistors,” in International Symposium on VLSI Technology, Systems and Application (IEEE Press, 2016).
(33)
For a much better explanation of quantum computation, see Scott Aaronson, Quantum Computing since Democritus (Cambridge University Press, 2013).
(34)
The paper that established a clear complexity-theoretic distinction between classical and quantum computation: Ethan Bernstein and Umesh Vazirani, “Quantum complexity theory,” SIAM Journal on Computing 26 (1997): 1411–73.
(35)
The following article by a renowned physicist provides a good introduction to the current state of understanding and technology: John Preskill, “Quantum computing in the NISQ era and beyond,” arXiv:1801.00862 (2018).
(36)
On the maximum computational ability of a one-kilogram object: Seth Lloyd, “Ultimate physical limits to computation,” Nature 406 (2000): 1047–54.
(37)
For an example of the suggestion that humans may be the pinnacle of physically achievable intelligence, see Kevin Kelly, “The myth of a superhuman AI,” Wired, April 25, 2017: “We tend to believe that the limit is way beyond us, way ‘above’ us, as we are ‘above’ an ant … What evidence do we have that the limit is not us?”
(38)
In case you are wondering about a simple trick to solve the halting problem: the obvious method of just running the program to see if it finishes doesn’t work, because that method doesn’t necessarily finish. You might wait a million years and still not know if the program is really stuck in an infinite loop or just taking its time.
(39)
The proof that the halting problem is undecidable is an elegant piece of trickery. The question: Is there a LoopChecker program that, for any program P and any input X, decides correctly, in finite time, whether P applied to input X will halt and produce a result or keep chugging away forever? Suppose that LoopChecker exists. Now write a program Q that calls LoopChecker as a subroutine, with Q itself and X as inputs, and then does the opposite of what LoopChecker predicts. So, if LoopChecker says that Q halts, Q doesn’t halt, and vice versa. Thus, the assumption that LoopChecker exists leads to a contradiction, so LoopChecker cannot exist.
(40)
I say “appear” because, as yet, the claim that the class of NP-complete problems requires superpolynomial time (usually referred to as P ≠ NP) is still an unproven conjecture. After almost fifty years of research, however, nearly all mathematicians and computer scientists are convinced the claim is true.
(41)
Lovelace’s writings on computation appear mainly in her notes attached to her translation of an Italian engineer’s commentary on Babbage’s engine: L. F. Menabrea, “Sketch of the Analytical Engine invented by Charles Babbage,” trans. Ada, Countess of Lovelace, in Scientific Memoirs, vol. III, ed. R. Taylor (R. and J. E. Taylor, 1843). Menabrea’s original article, written in French and based on lectures given by Babbage in 1840, appears in Bibliothèque Universelle de Genève 82 (1842).
(42)
One of the seminal early papers on the possibility of artificial intelligence: Alan Turing, “Computing machinery and intelligence,” Mind 59 (1950): 433–60.
(43)
The Shakey project at SRI is summarized in a retrospective by one of its leaders: Nils Nilsson, “Shakey the robot,” technical note 323 (SRI International, 1984). A twentyfour-minute film, SHAKEY: Experimentation in Robot Learning and Planning, was made in 1969 and garnered national attention.
(44)
The book that marked the beginning of modern, probability-based AI: Judea Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).
(45)
Technically, chess is not fully observable. A program does need to remember a small amount of information to determine the legality of castling and en passant moves and to define draws by repetition or by the fifty-move rule.
(46)
For a complete exposition, see Chapter 2 of Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd ed. (Pearson, 2010).
(47)
The size of the state space for StarCraft is discussed by Santiago Ontañon et al., “A survey of real-time strategy game AI research and competition in StarCraft,” IEEE Transactions on Computational Intelligence and AI in Games 5 (2013): 293–311. Vast numbers of moves are possible because a player can move all units simultaneously. The numbers go down as restrictions are imposed on how many units or groups of units can be moved at once.
(48)
On human-machine competition in StarCraft: Tom Simonite, “DeepMind beats pros at StarCraft in another triumph for bots,” Wired, January 25, 2019.
(49)
AlphaZero is described by David Silver et al., “Mastering chess and shogi by self-play with a general reinforcement learning algorithm,” arXiv:1712.01815 (2017).
(50)
Optimal paths in graphs are found using the A* algorithm and its many descendants: Peter Hart, Nils Nilsson, and Bertram Raphael, “A formal basis for the heuristic determination of minimum cost paths,” IEEE Transactions on Systems Science and Cybernetics SSC-4 (1968): 100–107.
(51)
The paper that introduced the Advice Taker program and logic-based knowledge systems: John McCarthy, “Programs with common sense,” in Proceedings of the Symposium on Mechanisation of Thought Processes (Her Majesty’s Stationery Office, 1958).
(52)
To get some sense of the significance of knowledge-based systems, consider database systems. A database contains concrete, individual facts, such as the location of my keys and the identities of your Facebook friends. Database systems cannot store general rules, such as the rules of chess or the legal definition of British citizenship. They can count how many people called Alice have friends called Bob, but they cannot determine whether a particular Alice meets the conditions for British citizenship or whether a particular sequence of moves on a chessboard will lead to checkmate. Database systems cannot combine two pieces of knowledge to produce a third: they support memory but not reasoning. (It is true that many modern database systems provide a way to add rules and a way to use those rules to derive new facts; to the extent that they do, they are really knowledge-based systems.) Despite being highly constricted versions of knowledge-based systems, database systems underlie most of present-day commercial activity and generate hundreds of billions of dollars in value every year.
(53)
The original paper describing the completeness theorem for first-order logic: Kurt Gödel, “Die Vollständigkeit der Axiome des logischen Funktionenkalküls,” Monatshefte für Mathematik 37 (1930): 349–60.
(54)
The reasoning algorithm for first-order logic does have a gap: if there is no answer — that is, if the available knowledge is insufficient to give an answer either way — then the algorithm may never finish. This is unavoidable: it is mathematically impossible for a correct algorithm always to terminate with “don’t know,” for essentially the same reason that no algorithm can solve the halting problem (page 37).
(55)
The first algorithm for theorem-proving in first-order logic worked by reducing firstorder sentences to (very large numbers of) propositional sentences: Martin Davis and Hilary Putnam, “A computing procedure for quantification theory,” Journal of the ACM 7 (1960): 201–15. Robinson’s resolution algorithm operated directly on first-order logical sentences, using “unification” to match complex expressions containing logical variables: J. Alan Robinson, “A machine-oriented logic based on the resolution principle,” Journal of the ACM 12 (1965): 23–41.
(56)
One might wonder how Shakey the logical robot ever reached any definite conclusions about what to do. The answer is simple: Shakey’s knowledge base contained false assertions. For example, Shakey believed that by executing “push object A through door D into room B,” object A would end up in room B. This belief was false because Shakey could get stuck in the doorway or miss the doorway altogether or someone might sneakily remove object A from Shakey’s grasp. Shakey’s plan execution module could detect plan failure and replan accordingly, so Shakey was not, strictly speaking, a purely logical system.
(57)
An early commentary on the role of probability in human thinking: Pierre-Simon Laplace, Essai philosophique sur les probabilités (Mme. Ve. Courcier, 1814).
(58)
Bayesian logic described in a fairly nontechnical way: Stuart Russell, “Unifying logic and probability,” Communications of the ACM 58 (2015): 88–97. The paper draws heavily on the PhD thesis research of my former student Brian Milch.
(59)
The original source for Bayes’ theorem: Thomas Bayes and Richard Price, “An essay towards solving a problem in the doctrine of chances,” Philosophical Transactions of the Royal Society of London 53 (1763): 370–418.
(60)
Technically, Samuel’s program did not treat winning and losing as absolute rewards; by fixing the value of material to be positive; however, the program generally tended to work towards winning.
(61)
The application of reinforcement learning to produce a world-class backgammon program: Gerald Tesauro, “Temporal difference learning and TD-Gammon,” Communications of the ACM 38 (1995): 58–68.
(62)
The DQN system that learns to play a wide variety of video games using deep RL: Volodymyr Mnih et al., “Human-level control through deep reinforcement learning,” Nature 518 (2015): 529–33.
(63)
Bill Gates’s remarks on Dota 2 AI: Catherine Clifford, “Bill Gates says gamer bots from Elon Musk-backed nonprofit are ‘huge milestone’ in A.I.,” CNBC, June 28, 2018.
(64)
An account of OpenAI Five’s victory over the human world champions at Dota 2: Kelsey Piper, “AI triumphs against the world’s top pro team in strategy game Dota 2,” Vox, April 13, 2019.
(65)
A compendium of cases in the literature where misspecification of reward functions led to unexpected behavior: Victoria Krakovna, “Specification gaming examples in AI,” Deep Safety (blog), April 2, 2018.
(66)
A case where an evolutionary fitness function defined in terms of maximum velocity led to very unexpected results: Karl Sims, “Evolving virtual creatures,” in Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1994).
(67)
For a fascinating exposition of the possibilities of reflex agents, see Valentino Braitenberg, Vehicles: Experiments in Synthetic Psychology (MIT Press, 1984).
(68)
News article on a fatal accident involving a vehicle in autonomous mode that hit a pedestrian: Devin Coldewey, “Uber in fatal crash detected pedestrian but had emergency braking disabled,” TechCrunch, May 24, 2018.
(69)
On steering control algorithms, see, for example, Jarrod Snider, “Automatic steering methods for autonomous automobile path tracking,” technical report CMU-RI-TR-09-08, Robotics Institute, Carnegie Mellon University, 2009.
(70)
Norfolk and Norwich terriers are two categories in the ImageNet database. They are notoriously hard to tell apart and were viewed as a single breed until 1964.
(71)
A very unfortunate incident with image labeling: Daniel Howley, “Google Photos mislabels 2 black Americans as gorillas,” Yahoo Tech, June 29, 2015.
(72)
Follow-up article on Google and gorillas: Tom Simonite, “When it comes to gorillas, Google Photos remains blind,” Wired, January 11, 2018.

الفصل الثالث: كيف قد يتطوَّر الذكاء الاصطناعي في المُستقبل؟

(1)
The basic plan for game-playing algorithms was laid out by Claude Shannon, “Programming a computer for playing chess,” Philosophical Magazine, 7th ser., 41 (1950): 256–75.
(2)
See figure 5٫12 of Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 1st ed. (Prentice Hall, 1995). Note that the rating of chess players and chess programs is not an exact science. Kasparov’s highest-ever Elo rating was 2851, achieved in 1999, but current chess engines such as Stockfish are rated at 3300 or more.
(3)
The earliest reported autonomous vehicle on a public road: Ernst Dickmanns and Alfred Zapp, “Autonomous high speed road vehicle guidance by computer vision,” IFAC Proceedings Volumes 20 (1987): 221–26.
(4)
The safety record for Google (subsequently Waymo) vehicles: “Waymo safety report: On the road to fully self-driving,” 2018.
(5)
So far there have been at least two driver fatalities and one pedestrian fatality. Some references follow, along with brief quotes describing what happened. Danny Yadron and Dan Tynan, “Tesla driver dies in first fatal crash while using autopilot mode,” Guardian, June 30, 2016: “The autopilot sensors on the Model S failed to distinguish a white tractor-trailer crossing the highway against a bright sky.” Megan Rose Dickey, “Tesla Model X sped up in Autopilot mode seconds before fatal crash, according to NTSB,” TechCrunch, June 7, 2018: “At 3 seconds prior to the crash and up to the time of impact with the crash attenuator, the Tesla’s speed increased from 62 to 70٫8 mph, with no precrash braking or evasive steering movement detected.” Devin Coldewey, “Uber in fatal crash detected pedestrian but had emergency braking disabled,” TechCrunch, May 24, 2018: “Emergency braking maneuvers are not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle behavior.”
(6)
The Society of Automotive Engineers (SAE) defines six levels of automation, where Level 0 is none at all and Level 5 is full automation: “The full-time performance by an automatic driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver.”
(7)
Forecast of economic effects of automation on transportation costs: Adele Peters, “It could be 10 times cheaper to take electric robo-taxis than to own a car by 2030,” Fast Company, May 30, 2017.
(8)
The impact of accidents on the prospects for regulatory action on autonomous vehicles: Richard Waters, “Self-driving car death poses dilemma for regulators,” Financial Times, March 20, 2018.
(9)
The impact of accidents on public perception of autonomous vehicles: Cox Automotive, “Autonomous vehicle awareness rising, acceptance declining, according to Cox Automotive mobility study,” August 16, 2018.
(10)
The original chatbot: Joseph Weizenbaum, “ELIZA — a computer program for the study of natural language communication between man and machine,” Communications of the ACM 9 (1966): 36–45.
(11)
See physiome.org for current activities in physiological modeling. Work in the 1960s assembled models with thousands of differential equations: Arthur Guyton, Thomas Coleman, and Harris Granger, “Circulation: Overall regulation,” Annual Review of Physiology 34 (1972): 13–44.
(12)
Some of the earliest work on tutoring systems was done by Pat Suppes and colleagues at Stanford: Patrick Suppes and Mona Morningstar, “Computer-assisted instruction,” Science 166 (1969): 343–50.
(13)
Michael Yudelson, Kenneth Koedinger, and Geoffrey Gordon, “Individualized Bayesian knowledge tracing models,” in Artificial Intelligence in Education: 16th International Conference, ed. H. Chad Lane et al. (Springer, 2013).
(14)
For an example of machine learning on encrypted data, see, for example, Reza Shokri and Vitaly Shmatikov, “Privacy-preserving deep learning,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (ACM, 2015).
(15)
A retrospective on the first smart home, based on a lecture by its inventor, James Sutherland: James E. Tomayko, “Electronic Computer for Home Operation (ECHO): The first home computer,” IEEE Annals of the History of Computing 16 (1994): 59–61.
(16)
Summary of a smart-home project based on machine learning and automated decisions: Diane Cook et al., “MavHome: An agent-based smart home,” in Proceedings of the 1st IEEE International Conference on Pervasive Computing and Communications (IEEE, 2003).
(17)
For the beginnings of an analysis of user experiences in smart homes, see Scott Davidoff et al., “Principles of smart home control,” in Ubicomp 2006: Ubiquitous Computing, ed. Paul Dourish and Adrian Friday (Springer, 2006).
(18)
Commercial announcement of AI-based smart homes: “The Wolff Company unveils revolutionary smart home technology at new Annadel Apartments in Santa Rosa, California,” Business Insider, March 12, 2018.
(19)
Article on robot chefs as commercial products: Eustacia Huen, “The world’s first home robotic chef can cook over 100 meals,” Forbes, October 31, 2016.
(20)
Report from my Berkeley colleagues on deep RL for robotic motor control: Sergey Levine et al., “End-to-end training of deep visuomotor policies,” Journal of Machine Learning Research 17 (2016): 1–40.
(21)
On the possibilities for automating the work of hundreds of thousands of warehouse workers: Tom Simonite, “Grasping robots compete to rule Amazon’s warehouses,” Wired, July 26, 2017.
(22)
I’m assuming a generous one laptop-CPU minute per page, or about 1011 operations. A third-generation tensor processing unit from Google runs at about 1017 operations per second, meaning that it can read a million pages per second, or about five hours for eighty million two-hundred-page books.
(23)
A 2003 study on the global volume of information production by all channels: Peter Lyman and Hal Varian, “How much information?” sims.berkeley.edu/research/projects/how-much-info-2003.
(24)
For details on the use of speech recognition by intelligence agencies, see Dan Froomkin, “How the NSA converts spoken words into searchable text,” The Intercept, May 5, 2015.
(25)
Analysis of visual imagery from satellites is an enormous task: Mike Kim, “Mapping poverty from space with the World Bank,” Medium.com, January 4, 2017. Kim estimates eight million people working 24/7, which converts to more than thirty million people working forty hours per week. I suspect this is an overestimate in practice, because the vast majority of the images would exhibit negligible change over the course of one day. On the other hand, the US intelligence community employs tens of thousands of people sitting in vast rooms staring at satellite images just to keep track of what’s happening in small regions of interest; so one million people is probably about right for the whole world.
(26)
There is substantial progress towards a global observatory based on real-time satellite image data: David Jensen and Jillian Campbell, “Digital earth: Building, financing and governing a digital ecosystem for planetary data,” white paper for the UN Science-Policy-Business Forum on the Environment, 2018.
(27)
Luke Muehlhauser has written extensively on AI predictions, and I am indebted to him for tracking down original sources for the quotations that follow. See Luke Muehlhauser, “What should we learn from past AI forecasts?” Open Philanthropy Project report, 2016.
(28)
A forecast of the arrival of human-level AI within twenty years: Herbert Simon, The New Science of Management Decision (Harper & Row, 1960).
(29)
A forecast of the arrival of human-level AI within a generation: Marvin Minsky, Computation: Finite and Infinite Machines (Prentice Hall, 1967).
(30)
John McCarthy’s forecast of the arrival of human-level AI within “five to 500 years”: Ian Shenker, “Brainy robots in our future, experts think,” Detroit Free Press, September 30, 1977.
(31)
For a summary of surveys of AI researchers on their estimates for the arrival of humanlevel AI, see aiimpacts.org. An extended discussion of survey results on human-level AI is given by Katja Grace et al., “When will AI exceed human performance? Evidence from AI experts,” arXiv:1705.08807v3 (2018).
(32)
For a chart mapping raw computer power against brain power, see Ray Kurzweil, “The law of accelerating returns,” Kurzweilai.net, March 7, 2001.
(33)
The Allen Institute’s Project Aristo: allenai.org/aristo.
(34)
For an analysis of the knowledge required to perform well on fourth-grade tests of comprehension and common sense, see Peter Clark et al., “Automatic construction of inference-supporting knowledge bases,” in Proceedings of the Workshop on Automated Knowledge Base Construction (2014), akbc.ws/2014.
(35)
The NELL project on machine reading is described by Tom Mitchell et al., “Neverending learning,” Communications of the ACM 61 (2018): 103–15.
(36)
The idea of bootstrapping inferences from text is due to Sergey Brin, “Extracting patterns and relations from the World Wide Web,” in The World Wide Web and Databases, ed. Paolo Atzeni, Alberto Mendelzon, and Giansalvatore Mecca (Springer, 1998).
(37)
For a visualization of the black-hole collision detected by LIGO, see LIGO Lab Caltech, “Warped space and time around colliding black holes,” February 11, 2016, youtube.com/watch?v=1agm33iEAuo.
(38)
The first publication describing observation of gravitational waves: Abbott et al., “Observation of gravitational waves from a binary black hole Physical Review Letters 116 (2016): 061102.
(39)
On babies as scientists: Alison Gopnik, Andrew Meltzoff, Patricia Kuhl, The Scientist in the Crib: Minds, Brains, and How Children Learn (William Morrow, 1999).
(40)
A summary of several projects on automated scientific analysis of experimental data to discover laws: Patrick Langley et al., Scientific Discovery: Computational Explorations of the Creative Processes (MIT Press, 1987).
(41)
Some early work on machine learning guided by prior knowledge: Stuart Russell, The Use of Knowledge in Analogy and Induction (Pitman, 1989).
(42)
Goodman’s philosophical analysis of induction remains a source of inspiration: Nelson Goodman, Fact, Fiction, and Forecast (University of London Press, 1954).
(43)
A veteran AI researcher complains about mysticism in the philosophy of science: Herbert Simon, “Explaining the ineffable: AI on the topics of intuition, insight and inspiration,” in Proceedings of the 14th International Conference on Artificial Intelligence, ed. Chris Mellish (Morgan Kaufmann, 1995).
(44)
A survey of inductive logic programming by two originators of the field: Stephen Muggleton and Luc de Raedt, “Inductive logic programming: Theory and methods,” Journal of Logic Programming 19-20 (1994): 629–79.
(45)
For an early mention of the importance of encapsulating complex operations as new primitive actions, see Alfred North Whitehead, An Introduction to Mathematics (Henry Holt, 1911).
(46)
Work demonstrating that a simulated robot can learn entirely by itself to stand up: John Schulman et al., “High-dimensional continuous control using generalized advantage estimation,” arXiv:1506.02438 (2015). A video demonstration is available at youtube.com/watch?v=SHLuf2ZBQSw.
(47)
A description of a reinforcement learning system that learns to play a capture-the-flag video game: Max Jaderberg et al., “Human-level performance in first-person multiplayer games with population-based deep reinforcement learning,” arXiv:1807.01281 (2018).
(48)
A view of AI progress over the next few years: Peter Stone et al., “Artificial intelligence and life in 2030,” One Hundred Year Study on Artificial Intelligence, report of the 2015 Study Panel, 2016.
(49)
The media-fueled argument between Elon Musk and Mark Zuckerberg: Peter Holley, “Billionaire burn: Musk says Zuckerberg’s understanding of AI threat ‘is limited,’” The Washington Post, July 25, 2017.
(50)
On the value of search engines to individual users: Erik Brynjolfsson, Felix Eggers, and Avinash Gannamaneni, “Using massive online choice experiments to measure changes in well-being,” working paper no. 24514, National Bureau of Economic Research, 2018.
(51)
Penicillin was discovered several times and its curative powers were described in medical publications, but no one seems to have noticed. See en.wikipedia.org/wiki/History_of_penicillin.
(52)
For a discussion of some of the more esoteric risks from omniscient, clairvoyant AI systems, see David Auerbach, “The most terrifying thought experiment of all time,” Slate, July 17, 2014.
(53)
An analysis of some potential pitfalls in thinking about advanced AI: Kevin Kelly, “The myth of a superhuman AI,” Wired, April 25, 2017.
(54)
Machines may share some aspects of cognitive structure with humans, particularly those aspects dealing with perception and manipulation of the physical world and the conceptual structures involved in natural language understanding. Their deliberative processes are likely to be quite different because of the enormous disparities in hardware.
(55)
According to 2016 survey data, the eighty-eighth percentile corresponds to $100,000 per year: American Community Survey, US Census Bureau, www.census.gov/programs-surveys/acs. For the same year, global per capita GDP was $10,133: National Accounts Main Aggregates Database, UN Statistics Division, unstats.un.org/unsd/snaama.
(56)
If the GDP growth phases in over ten years or twenty years, it’s worth $9,400 trillion or $6,800 trillion, respectively — still nothing to sneeze at. On an interesting historical note, I. J. Good, who popularized the notion of an intelligence explosion (page 142), estimated the value of human-level AI to be at least “one megaKeynes,” referring to the fabled economist John Maynard Keynes. The value of Keynes’s contributions was estimated in 1963 as £100 billion, so a megaKeynes comes out to around $2,200,000 trillion in 2016 dollars. Good pinned the value of AI primarily on its potential to ensure that the human race survives indefinitely. Later, he came to wonder whether he should have added a minus sign.
(57)
The EU announced plans for $24 billion in research and development spending for the period 2019–20. See European Commission, “Artificial intelligence: Commission outlines a European approach to boost investment and set ethical guidelines,” press release, April 25, 2018. China’s long-term investment plan for AI, announced in 2017, envisages a core AI industry generating $150 billion annually by 2030. See, for example, Paul Mozur, “Beijing wants A.I. to be made in China by 2030,” The New York Times, July 20, 2017.
(58)
See, for example, Rio Tinto’s Mine of the Future program at riotinto.com/australia/pilbara/mine-of-the-future-9603.aspx.
(59)
A retrospective analysis of economic growth: Jan Luiten van Zanden et al., eds., How Was Life? Global Well-Being since 1820 (OECD Publishing, 2014).
(60)
The desire for relative advantage over others, rather than an absolute quality of life, is a positional good; see Chapter 9.

الفصل الرابع: إساءة استخدام الذكاء الاصطناعي

(1)
Wikipedia’s article on the Stasi has several useful references on its workforce and its overall impact on East German life.
(2)
For details on Stasi files, see Cullen Murphy, God’s Jury: The Inquisition and the Making of the Modern World (Houghton Mifflin Harcourt, 2012).
(3)
For a thorough analysis of AI surveillance systems, see Jay Stanley, The Dawn of Robot Surveillance (American Civil Liberties Union, 2019).
(4)
Recent books on surveillance and control include Shoshana Zuboff, The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (PublicAffairs, 2019) and Roger McNamee, Zucked: Waking Up to the Facebook Catastrophe (Penguin Press, 2019).
(5)
News article on a blackmail bot: Avivah Litan, “Meet Delilah — the first insider threat Trojan,” Gartner Blog Network, July 14, 2016.
(6)
For a low-tech version of human susceptibility to misinformation, in which an unsuspecting individual becomes convinced that the world is being destroyed by meteor strikes, see Derren Brown: Apocalypse, “Part One,” directed by Simon Dinsell, 2012, youtube.com/watch?v=o_CUrMJOxqs.
(7)
An economic analysis of reputation systems and their corruption is given by Steven Tadelis, “Reputation and feedback systems in online platform markets,” Annual Review of Economics 8 (2016): 321–40.
(8)
Goodhart’s law: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” For example, there may once have been a correlation between faculty quality and faculty salary, so the US News & World Report college rankings measure faculty quality by faculty salaries. This has contributed to a salary arms race that benefits faculty members but not the students who pay for those salaries. The arms race changes faculty salaries in a way that does not depend on faculty quality, so the correlation tends to disappear.
(9)
An article describing German efforts to police public discourse: Bernhard Rohleder, “Germany set out to delete hate speech online. Instead, it made things worse,” World-Post, February 20, 2018.
(10)
On the “infopocalypse”: Aviv Ovadya, “What’s worse than fake news? The distortion of reality itself,” WorldPost, February 22, 2018.
(11)
On the corruption of online hotel reviews: Dina Mayzlin, Yaniv Dover, and Judith Chevalier, “Promotional reviews: An empirical investigation review manipulation,” American Economic Review 104 (2014): 2421–55.
(12)
Statement of Germany at the Meeting of the Group of Governmental Experts, Convention on Certain Conventional Weapons, Geneva, April 10, 2018.
(13)
The Slaughterbots movie, funded by the Future of Life Institute, appeared in November 2017 and is available at youtube.com/watch?v=9CO6M2HsoIA.
(14)
For a report on one of the bigger faux pas in military public relations, see Dan Lamothe, “Pentagon agency wants drones to hunt in packs, like wolves,” The Washington Post, January 23, 2015.
(15)
Announcement of a large-scale drone swarm experiment: US Department of Defense, “Department of Defense announces successful micro-drone demonstration,” news release no. NR-008-17, January 9, 2017.
(16)
Examples of research centers studying the impact of technology on employment are the Work and Intelligent Tools and Systems group at Berkeley, the Future of Work and Workers project at the Center for Advanced Study in the Behavioral Sciences at Stanford, and the Future of Work Initiative at Carnegie Mellon University.
(17)
A pessimistic take on future technological unemployment: Martin Ford, Rise of the Robots: Technology and the Threat of a Jobless Future (Basic Books, 2015).
(18)
Calum Chace, The Economic Singularity: Artificial Intelligence and the Death of Capitalism (Three Cs, 2016).
(19)
For an excellent collection of essays, see Ajay Agrawal, Joshua Gans, and Avi Goldfarb, eds., The Economics of Artificial Intelligence: An Agenda (National Bureau of Economic Research, 2019).
(20)
The mathematical analysis behind this “inverted-U” employment curve is given by James Bessen, “Artificial intelligence and jobs: The role of demand” in The Economics of Artificial Intelligence, ed. Agrawal, Gans, and Goldfarb.
(21)
For a discussion of economic dislocation arising from automation, see Eduardo Porter, “Tech is splitting the US work force in two,” The New York Times, February 4, 2019. The article cites the following report for this conclusion: David Autor and Anna Salomons, “Is automation labor-displacing? Productivity growth, employment, and the labor share,” Brookings Papers on Economic Activity (2018).
(22)
For data on the growth of banking in the twentieth century, see Thomas Philippon, “The evolution of the US financial industry from 1860 to 2007: Theory and evidence,” working paper, 2008.
(23)
The bible for jobs data and the growth and decline of occupations: US Bureau of Labor Statistics, Occupational Outlook Handbook: 2018–2019 Edition (Bernan Press, 2018).
(24)
A report on trucking automation: Lora Kolodny, “Amazon is hauling cargo in selfdriving trucks developed by Embark,” CNBC, January 30, 2019.
(25)
The progress of automation in legal analytics, describing the results of a contest: Jason Tashea, “AI software is more accurate, faster than attorneys when assessing NDAs,” ABA Journal, February 26, 2018.
(26)
A commentary by a distinguished economist, with a title explicitly evoking Keynes’s 1930 article: Lawrence Summers, “Economic possibilities for our children,” NBER Reporter (2013).
(27)
The analogy between data science employment and a small lifeboat for a giant cruise ship comes from a discussion with Yong Ying-I, head of Singapore’s Public Service Division. She conceded that it was correct on the global scale, but noted that “Singapore is small enough to fit in the lifeboat.”
(28)
Support for UBI from a conservative viewpoint: Sam Bowman, “The ideal welfare system is a basic income,” Adam Smith Institute, November 25, 2013.
(29)
Support for UBI from a progressive viewpoint: Jonathan Bartley, “The Greens endorse a universal basic income. Others need to follow,” The Guardian, June 2, 2017.
(30)
Chace, in The Economic Singularity, calls the “paradise” version of UBI the Star Trek economy, noting that in the more recent series of Star Trek episodes, money has been abolished because technology has created essentially unlimited material goods and energy. He also points to the massive changes in economic and social organization that will be needed to make such a system successful.
(31)
The economist Richard Baldwin also predicts a future of personal services in his book The Globotics Upheaval: Globalization, Robotics, and the Future of Work (Oxford University Press, 2019).
(32)
The book that is viewed as having exposed the failure of “whole-word” literacy education and launched decades of struggle between the two main schools of thought on reading: Rudolf Flesch, Why Johnny Can’t Read: And What You Can Do about It (Harper & Bros., 1955).
(33)
On educational methods that enable the recipient to adapt to the rapid rate of technological and economic change in the next few decades: Joseph Aoun, Robot-Proof: Higher Education in the Age of Artificial Intelligence (MIT Press, 2017).
(34)
A radio lecture in which Turing predicted that humans would be overtaken by machines: Alan Turing, “Can digital machines think?,” May 15, 1951, radio broadcast, BBC Third Programme. Typescript available at turingarchive.org.
(35)
News article describing the “naturalization” of Sophia as a citizen of Saudi Arabia: Dave Gershgorn, “Inside the mechanical brain of the world’s first robot citizen,” Quartz, November 12, 2017.
(36)
On Yann LeCun’s view of Sophia: Shona Ghosh, “Facebook’s AI boss described Sophia the robot as ‘complete b——t’ and ‘Wizard-of-Oz AI,’” Business Insider, January 6, 2018.
(37)
An EU proposal on legal rights for robots: Committee on Legal Affairs of the European Parliament, “Report with recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(INL)),” 2017.
(38)
The GDPR provision on a “right to an explanation” is not, in fact, new: it is very similar to Article 15(1) of the 1995 Data Protection Directive, which it supersedes.
(39)
Here are three recent papers providing insightful mathematical analyses of fairness: Moritz Hardt, Eric Price, and Nati Srebro, “Equality of opportunity in supervised learning,” in Advances in Neural Information Processing Systems 29, ed. Daniel Lee et al. (2016); Matt Kusner et al., “Counterfactual fairness,” in Advances in Neural Information Processing Systems 30, ed. Isabelle Guyon et al. (2017); Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, “Inherent trade-offs in the fair determination of risk scores,” in 8th Innovations in Theoretical Computer Science Conference, ed. Christos Papadimitriou (Dagstuhl Publishing, 2017).
(40)
News article describing the consequences of software failure for air traffic control: Simon Calder, “Thousands stranded by flight cancellations after systems failure at Europe’s air-traffic coordinator,” The Independent, April 3, 2018.

الفصل الخامس: الذكاء الاصطناعي الفائق الذكاء

(1)
Lovelace wrote, “The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform. It can follow analysis; but it has no power of anticipating any analytical relations or truths.” This was one of the arguments against AI that was refuted by Alan Turing, “Computing machinery and intelligence,” Mind 59 (1950): 433–60.
(2)
The earliest known article on existential risk from AI was by Richard Thornton, “The age of machinery,” Primitive Expounder IV (1847): 281.
(3)
“The Book of the Machines” was based on an earlier article by Samuel Butler, “Darwin among the machines,” The Press (Christchurch, New Zealand), June 13, 1863.
(4)
Another lecture in which Turing predicted the subjugation of humankind: Alan Turing, “Intelligent machinery, a heretical theory” (lecture given to the 51 Society, Manchester, 1951). Typescript available at turingarchive.org.
(5)
Wiener’s prescient discussion of technological control over humanity and a plea to retain human autonomy: Norbert Wiener, The Human Use of Human Beings (Riverside Press, 1950).
(6)
The front-cover blurb from Wiener’s 1950 book is remarkably similar to the motto of the Future of Life Institute, an organization dedicated to studying the existential risks that humanity faces: “Technology is giving life the potential to flourish like never before … or to self-destruct.”
(7)
An updating of Wiener’s views arising from his increased appreciation of the possibility of intelligent machines: Norbert Wiener, God and Golem, Inc.: A Comment on Certain Points Where Cybernetics Impinges on Religion (MIT Press, 1964).
(8)
Asimov’s Three Laws of Robotics first appeared in Isaac Asimov, “Runaround,” Astounding Science Fiction, March 1942. The laws are as follows:
(1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.
(2) A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
(3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.
It is important to understand that Asimov proposed these laws as a way to generate interesting story plots, not as a serious guide for future roboticists. Several of his stories, including “Runaround,” illustrate the problematic consequences of taking the laws literally. From the standpoint of modern AI, the laws fail to acknowledge any element of probability and risk: the legality of robot actions that expose a human to some probability of harm — however infinitesimal — is therefore unclear.
(9)
The notion of instrumental goals is due to Stephen Omohundro, “The nature of selfimproving artificial intelligence” (unpublished manuscript, 2008). See also Stephen Omohundro, “The basic AI drives,” in Artificial General Intelligence 2008: Proceedings of the First AGI Conference, ed. Pei Wang, Ben Goertzel, and Stan Franklin (IOS Press, 2008).
(10)
The objective of Johnny Depp’s character, Will Caster, seems to be to solve the problem of physical reincarnation so that he can be reunited with his wife, Evelyn. This just goes to show that the nature of the overarching objective doesn’t matter — the instrumental goals are all the same.
(11)
The original source for the idea of an intelligence explosion: I. J. Good, “Speculations concerning the first ultraintelligent machine,” in Advances in Computers, vol. 6, ed. Franz Alt and Morris Rubinoff (Academic Press, 1965).
(12)
An example of the impact of the intelligence explosion idea: Luke Muehlhauser, in Facing the Intelligence Explosion (intelligenceexplosion.com), writes, “Good’s paragraph ran over me like a train.”
(13)
Diminishing returns can be illustrated as follows: suppose that a 16 percent improvement in intelligence creates a machine capable of making an 8 percent improvement, which in turn creates a 4 percent improvement, and so on. This process reaches a limit at about 36 percent above the original level. For more discussion on these issues, see Eliezer Yudkowsky, “Intelligence explosion microeconomics,” technical report 2013-1, Machine Intelligence Research Institute, 2013.
(14)
For a view of AI in which humans become irrelevant, see Hans Moravec, Mind Children: The Future of Robot and Human Intelligence (Harvard University Press, 1988). See also Hans Moravec, Robot: Mere Machine to Transcendent Mind (Oxford University Press, 2000).

الفصل السادس: الجدل غير الواسع الدائر حول الذكاء الاصطناعي

(1)
A serious publication provides a serious review of Bostrom’s Superintelligence: Paths, Dangers, Strategies: “Clever cogs,” Economist, August 9, 2014.
(2)
A discussion of myths and misunderstandings concerning the risks of AI: Scott Alexander, “AI researchers on AI risk,” Slate Star Codex (blog), May 22, 2015.
(3)
The classic work on multiple dimensions of intelligence: Howard Gardner, Frames of Mind: The Theory of Multiple Intelligences (Basic Books, 1983).
(4)
On the implications of multiple dimensions of intelligence for the possibility of superhuman AI: Kevin Kelly, “The myth of a superhuman AI,” Wired, April 25, 2017.
(5)
Evidence that chimpanzees have better short-term memory than humans: Sana Inoue and Tetsuro Matsuzawa, “Working memory of numerals in chimpanzees,” Current Biology 17 (2007), R1004–5.
(6)
An important early work questioning the prospects for rule-based AI systems: Hubert Dreyfus, What Computers Can’t Do (MIT Press, 1972).
(7)
The first in a series of books seeking physical explanations for consciousness and raising doubts about the ability of AI systems to achieve real intelligence: Roger Penrose, The Emperor’s New Mind: Concerning Computers, Minds, and the Laws of Physics (Oxford University Press, 1989).
(8)
A revival of the critique of AI based on the incompleteness theorem: Luciano Floridi, “Should we be afraid of AI?” Aeon, May 9, 2016.
(9)
A revival of the critique of AI based on the Chinese room argument: John Searle, “What your computer can’t know,” The New York Review of Books, October 9, 2014.
(10)
A report from distinguished AI researchers claiming that superhuman AI is probably impossible: Peter Stone et al., “Artificial intelligence and life in 2030,” One Hundred Year Study on Artificial Intelligence, report of the 2015 Study Panel, 2016.
(11)
News article based on Andrew Ng’s dismissal of risks from AI: Chris Williams, “AI guru Ng: Fearing a rise of killer robots is like worrying about overpopulation on Mars,” Register, March 19, 2015.
(12)
An example of the “experts know best” argument: Oren Etzioni, “It’s time to intelligently discuss artificial intelligence,” Backchannel, December 9, 2014.
(13)
News article claiming that real AI researchers dismiss talk of risks: Erik Sofge, “Bill Gates fears AI, but AI researchers know better,” Popular Science, January 30, 2015.
(14)
Another claim that real AI researchers dismiss AI risks: David Kenny, “IBM’s open letter to Congress on artificial intelligence,” June 27, 2017, ibm.com/blogs/policy/kenny-artificial-intelligence-letter.
(15)
Report from the workshop that proposed voluntary restrictions on genetic engineering: Paul Berg et al., “Summary statement of the Asilomar Conference on Recombinant DNA Molecules,” Proceedings of the National Academy of Sciences 72 (1975): 1981–84.
(16)
Policy statement arising from the invention of CRISPR-Cas9 for gene editing: Organizing Committee for the International Summit on Human Gene Editing, “On human gene editing: International Summit statement,” December 3, 2015.
(17)
The latest policy statement from leading biologists: Eric Lander et al., “Adopt a moratorium on heritable genome editing,” Nature 567 (2019): 165–68.
(18)
Etzioni’s comment that one cannot mention risks if one does not also mention benefits appears alongside his analysis of survey data from AI researchers: Oren Etzioni, “No, the experts don’t think superintelligent AI is a threat to humanity,” MIT Technology Review, September 20, 2016. In his analysis he argues that anyone who expects superhuman AI to take more than twenty-five years — which includes this author as well as Nick Bostrom — is not concerned about the risks of AI.
(19)
A news article with quotations from the Musk-Zuckerberg “debate”: Alanna Petroff, “Elon Musk says Mark Zuckerberg’s understanding of AI is ‘limited,’” CNN Money, July 25, 2017.
(20)
In 2015 the Information Technology and Innovation Foundation organized a debate titled “Are super intelligent computers really a threat to humanity?” Robert Atkinson, director of the foundation, suggests that mentioning risks is likely to result in reduced funding for AI. Video available at itif.org/events/2015/06/30/are-super-intelligent-computers-really-threat-humanity; the relevant discussion begins at 41:30.
(21)
A claim that our culture of safety will solve the AI control problem without ever mentioning it: Steven Pinker, “Tech prophecy and the underappreciated causal power of ideas,” in Possible Minds: Twenty-Five Ways of Looking at AI, ed. John Brockman (Penguin Press, 2019).
(22)
For an interesting analysis of Oracle AI, see Stuart Armstrong, Anders Sandberg, and Nick Bostrom, “Thinking inside the box: Controlling and using an Oracle AI,” Minds and Machines 22 (2012): 299–324.
(23)
Views on why AI is not going to take away jobs: Kenny, “IBM’s open letter.”
(24)
An example of Kurzweil’s positive views of merging human brains with AI: Ray Kurzweil, interview by Bob Pisani, June 5, 2015, Exponential Finance Summit, New York, NY.
(25)
Article quoting Elon Musk on neural lace: Tim Urban, “Neuralink and the brain’s magical future,” Wait But Why, April 20, 2017.
(26)
For the most recent developments in Berkeley’s neural dust project, see David Piech et al., “StimDust: A 1٫7 mm3, implantable wireless precision neural stimulator with ultrasonic power and communication,” arXiv: 1807.07590 (2018).
(27)
Susan Schneider, in Artificial You: AI and the Future of Your Mind (Princeton University Press, 2019), points out the risks of ignorance in proposed technologies such as uploading and neural prostheses: that, absent any real understanding of whether electronic devices can be conscious and given the continuing philosophical confusion over persistent personal identity, we may inadvertently end our own conscious existences or inflict suffering on conscious machines without realizing that they are conscious.
(28)
An interview with Yann LeCun on AI risks: Guia Marie Del Prado, “Here’s what Facebook’s artificial intelligence expert thinks about the future,” Business Insider, September 23, 2015.
(29)
A diagnosis of AI control problems arising from an excess of testosterone: Steven Pinker, “Thinking does not imply subjugating,” in What to Think About Machines That Think, ed. John Brockman (Harper Perennial, 2015).
(30)
A seminal work on many philosophical topics, including the question of whether moral obligations may be perceived in the natural world: David Hume, A Treatise of Human Nature (John Noon, 1738).
(31)
An argument that a sufficiently intelligent machine cannot help but pursue human objectives: Rodney Brooks, “The seven deadly sins of AI predictions,” MIT Technology Review, October 6, 2017.
(32)
Pinker, “Thinking does not imply subjugating.”
(33)
For an optimistic view arguing that AI safety problems will necessarily be resolved in our favor: Steven Pinker, “Tech prophecy.”
(34)
On the unsuspected alignment between “skeptics” and “believers” in AI risk: Alexander, “AI researchers on AI risk.”

الفصل السابع: الذكاء الاصطناعي: توجُّه مُختلف

(1)
For a guide to detailed brain modeling, now slightly outdated, see Anders Sandberg and Nick Bostrom, “Whole brain emulation: A roadmap,” technical report 2008-3, Future of Humanity Institute, Oxford University, 2008.
(2)
For an introduction to genetic programming from a leading exponent, see John Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (MIT Press, 1992).
(3)
The parallel to Asimov’s Three Laws of Robotics is entirely coincidental.
(4)
The same point is made by Eliezer Yudkowsky, “Coherent extrapolated volition,” technical report, Singularity Institute, 2004. Yudkowsky argues that directly building in “Four Great Moral Principles That Are All We Need to Program into AIs” is a sure road to ruin for humanity. His notion of the “coherent extrapolated volition of humankind” has the same general flavor as the first principle; the idea is that a superintelligent AI system could work out what humans, collectively, really want.
(5)
You can certainly have preferences over whether a machine is helping you achieve your preferences or you are achieving them through your own efforts. For example, suppose you prefer outcome A to outcome B, all other things being equal. You are unable to achieve outcome A unaided, and yet you still prefer B to getting A with the machine’s help. In that case the machine should decide not to help you — unless perhaps it can do so in a way that is completely undetectable by you. You may, of course, have preferences about undetectable help as well as detectable help.
(6)
The phrase “the greatest good of the greatest number” originates in the work of Francis Hutcheson, An Inquiry into the Original of Our Ideas of Beauty and Virtue, In Two Treatises (D. Midwinter et al., 1725). Some have ascribed the formulation to an earlier comment by Wilhelm Leibniz; see Joachim Hruschka, “The greatest happiness principle and other early German anticipations of utilitarian theory,” Utilitas 3 (1991): 165–77.
(7)
One might propose that the machine should include terms for animals as well as humans in its own objective function. If these terms have weights that correspond to how much people care about animals, then the end result will be the same as if the machine cares about animals only through caring about humans who care about animals. Giving each living animal equal weight in the machine’s objective function would certainly be catastrophic — for example, we are outnumbered fifty thousand to one by Antarctic krill and a billion trillion to one by bacteria.
(8)
The moral philosopher Toby Ord made the same point to me in his comments on an early draft of this book: “Interestingly, the same is true in the study of moral philosophy. Uncertainty about moral value of outcomes was almost completely neglected in moral philosophy until very recently. Despite the fact that it is our uncertainty of moral matters that leads people to ask others for moral advice and, indeed, to do research on moral philosophy at all!”
(9)
One excuse for not paying attention to uncertainty about preferences is that it is formally equivalent to ordinary uncertainty, in the following sense: being uncertain about what I like is the same as being certain that I like likable things while being uncertain about what things are likable. This is just a trick that appears to move the uncertainty into the world, by making “likability by me” a property of objects rather than a property of me. In game theory, this trick has been thoroughly institutionalized since the 1960s, following a series of papers by my late colleague and Nobel laureate John Harsanyi: “Games with incomplete information played by ‘Bayesian’ players, Parts I–III,” Management Science 14 (1967, 1968): 159–82, 320–34, 486–502. In decision theory, the standard reference is the following: Richard Cyert and Morris de Groot, “Adaptive utility,” in Expected Utility Hypotheses and the Allais Paradox, ed. Maurice Allais and Ole Hagen (D. Reidel, 1979).
(10)
AI researchers working in the area of preference elicitation are an obvious exception. See, for example, Craig Boutilier, “On the foundations of expected expected utility,” in Proceedings of the 18th International Joint Conference on Artificial Intelligence (Morgan Kaufmann, 2003). Also Alan Fern et al., “A decision-theoretic model of assistance,” Journal of Artificial Intelligence Research 50 (2014): 71–104.
(11)
A critique of beneficial AI based on a misinterpretation of a journalist’s brief interview with the author in a magazine article: Adam Elkus, “How to be good: Why you can’t teach human values to artificial intelligence,” Slate, April 20, 2016.
(12)
The origin of trolley problems: Frank Sharp, “A study of the influence of custom on the moral judgment,” Bulletin of the University of Wisconsin 236 (1908).
(13)
The “anti-natalist” movement believes it is morally wrong for humans to reproduce because to live is to suffer and because humans’ impact on the Earth is profoundly negative. If you consider the existence of humanity to be a moral dilemma, then I suppose I do want machines to resolve this moral dilemma the right way.
(14)
Statement on China’s AI policy by Fu Ying, vice chair of the Foreign Affairs Committee of the National People’s Congress. In a letter to the 2018 World AI Conference in Shanghai, Chinese president Xi Jinping wrote, “Deepened international cooperation is required to cope with new issues in fields including law, security, employment, ethics and governance.” I am indebted to Brian Tse for bringing these statements to my attention.
(15)
A very interesting paper on the non-naturalistic non-fallacy, showing how preferences can be inferred from the state of the world as arranged by humans: Rohin Shah et al., “The implicit preference information in an initial state,” in Proceedings of the 7th International Conference on Learning Representations (2019), iclr.cc/Conferences/2019/Schedule.
(16)
Retrospective on Asilomar: Paul Berg, “Asilomar 1975: DNA modification secured,” Nature 455 (2008): 290–91.
(17)
News article reporting Putin’s speech on AI: “Putin: Leader in artificial intelligence will rule world,” Associated Press, September 4, 2017.

الفصل الثامن: الذكاء الاصطناعي النافع على نحو مثبت

(1)
Fermat’s Last Theorem asserts that the equation    has no solutions with a, b, and c being whole numbers and n being a whole number larger than 2. In the margin of his copy of Diophantus’s Arithmetica, Fermat wrote, “I have a truly marvellous proof of this proposition which this margin is too narrow to contain.” True or not, this guaranteed that mathematicians pursued a proof with vigor in the subsequent centuries. We can easily check particular cases — for example, is 73 equal to 63 + 53? (Almost, because 73 is 343 and 63 + 53 is 341, but “almost” doesn’t count.) There are, of course, infinitely many cases to check, and that’s why we need mathematicians and not just computer programmers.
(2)
A paper from the Machine Intelligence Research Institute poses many related issues: Scott Garrabrant and Abram Demski, “Embedded agency,” AI Alignment Forum, November 15, 2018.
(3)
The classic work on multiattribute utility theory: Ralph Keeney and Howard Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs (Wiley, 1976).
(4)
Paper introducing the idea of inverse RL: Stuart Russell, “Learning agents for uncertain environments,” in Proceedings of the 11th Annual Conference on Computational Learning Theory (ACM, 1998).
(5)
The original paper on structural estimation of Markov decision processes: Thomas Sargent, “Estimation of dynamic labor demand schedules under rational expectations,” Journal of Political Economy 86 (1978): 1009–44.
(6)
The first algorithms for IRL: Andrew Ng and Stuart Russell, “Algorithms for inverse reinforcement learning,” in Proceedings of the 17th International Conference on Machine Learning, ed. Pat Langley (Morgan Kaufmann, 2000).
(7)
Better algorithms for inverse RL: Pieter Abbeel and Andrew Ng, “Apprenticeship learning via inverse reinforcement learning,” in Proceedings of the 21st International Conference on Machine Learning, ed. Russ Greiner and Dale Schuurmans (ACM Press, 2004).
(8)
Understanding inverse RL as Bayesian updating: Deepak Ramachandran and Eyal Amir, “Bayesian inverse reinforcement learning,” in Proceedings of the 20th International Joint Conference on Artificial Intelligence, ed. Manuela Veloso (AAAI Press, 2007)
(9)
How to teach helicopters to fly and do aerobatic maneuvers: Adam Coates, Pieter Abbeel, and Andrew Ng, “Apprenticeship learning for helicopter control,” Communications of the ACM 52 (2009): 97–105.
(10)
The original name proposed for an assistance game was a cooperative inverse reinforcement learning game, or CIRL game. See Dylan Hadfield-Menell et al., “Cooperative inverse reinforcement learning,” in Advances in Neural Information Processing Systems 29, ed. Daniel Lee et al. (2016).
(11)
These numbers are chosen just to make the game interesting.
(12)
The equilibrium solution to the game can be found by a process called iterated best response: pick any strategy for Harriet; pick the best strategy for Robbie, given Harriet’s strategy; pick the best strategy for Harriet, given Robbie’s strategy; and so on. If this process reaches a fixed point, where neither strategy changes, then we have found a solution. The process unfolds as follows:
(1) Start with the greedy strategy for Harriet: make 2 paperclips if she prefers paperclips; make 1 of each if she is indifferent; make 2 staples if she prefers staples.
(2) There are three possibilities Robbie has to consider, given this strategy for Harriet:
(a) If Robbie sees Harriet make 2 paperclips, he infers that she prefers paperclips, so he now believes the value of a paperclip is uniformly distributed between 50 ȼ And $1٫00, with an average of 75 ȼ … In that case, his best plan is to make 90 paperclips with an expected value of $67٫50 for Harriet.
(b) If Robbie sees Harriet make 1 of each, he infers that she values paperclips and staples at 50 ȼ, so the best choice is to make 50 of each.
(c) If Robbie sees Harriet make 2 staples, then by the same argument as in 2(a), he should make 90 staples.
(3) Given this strategy for Robbie, Harriet’s best strategy is now somewhat different from the greedy strategy in step 1: if Robbie is going to respond to her making 1 of each by making 50 of each, then she is better off making 1 of each not just if she is exactly indifferent but if she is anywhere close to indifferent. In fact, the optimal policy is now to make 1 of each if she values paperclips anywhere between about 44٫6ȼ and 55٫4ȼ.
(4) Given this new strategy for Harriet, Robbie’s strategy remains unchanged. For example, if she chooses 1 of each, he infers that the value of a paperclip is uniformly distributed between 44٫6ȼ and 55٫4ȼ, with an average of 50ȼ, so the best choice is to make 50 of each. Because Robbie’s strategy is the same as in step 2, Harriet’s best response will be the same as in step 3, and we have found the equilibrium.
(13)
For a more complete analysis of the off-switch game, see Dylan Hadfield-Menell et al., “The off-switch game,” in Proceedings of the 26th International Joint Conference on Artificial Intelligence, ed. Carles Sierra (IJCAI, 2017).
(14)
The proof of the general result is quite simple if you don’t mind integral signs. Let P(u) be Robbie’s prior probability density over Harriet’s utility for the proposed action a. Then the value of going ahead with a is
(We will see shortly why the integral is split up in this way.) On the other hand, the value of action d, deferring to Harriet, is composed of two parts: if u > 0, then Harriet lets Robbie go ahead, so the value is u, but if u < 0, then Harriet switches Robbie off, so the value is 0:
Comparing the expressions for    and    we see immediately that    because the expression for    has the negative-utility region zeroed out. The two choices have equal value only when the negative region has zero probability — that is, when Robbie is already certain that Harriet likes the proposed action. The theorem is a direct analog of the well-known theorem concerning the non-negative expected value of information.
(15)
Perhaps the next elaboration in line, for the one human–one robot case, is to consider a Harriet who does not yet know her own preferences regarding some aspect of the world, or whose preferences have not yet been formed.
(16)
To see how exactly Robbie converges to an incorrect belief, consider a model in which Harriet is slightly irrational, making errors with a probability that diminishes exponentially as the size of error increases. Robbie offers Harriet 4 paperclips in return for 1 staple; she refuses. According to Robbie’s beliefs, this is irrational: even at 25 ȼ Per paperclip and 75 ȼ per staple, she should accept 4 for 1. Therefore, she must have made a mistake — but this mistake is much more likely if her true value is 25 ȼ than if it is, say, 30 ȼ, because the error costs her a lot more if her value for paperclips is 30ȼ … Now Robbie’s probability distribution has 25ȼ as the most likely value because it represents the smallest error on Harriet’s part, with exponentially lower probabilities for values higher than 25ȼ … If he keeps trying the same experiment, the probability distribution becomes more and more concentrated close to 25ȼ … In the limit, Robbie becomes certain that Harriet’s value for paperclips is 25ȼ.
(17)
Robbie could, for example, have a normal (Gaussian) distribution for his prior belief about the exchange rate, which stretches from −∞ to +∞.
(18)
For an example of the kind of mathematical analysis that may be needed, see Avrim Blum, Lisa Hellerstein, and Nick Littlestone, “Learning in the presence of finitely or infinitely many irrelevant attributes,” Journal of Computer and System Sciences 50 (1995): 32–40. Also Lori Dalton, “Optimal Bayesian feature selection,” in Proceedings of the 2013 IEEE Global Conference on Signal and Information Processing, ed. Charles Bouman, Robert Nowak, and Anna Scaglione (IEEE, 2013).
(19)
Here I am rephrasing slightly a question by Moshe Vardi at the Asilomar Conference on Beneficial AI, 2017.
(20)
Michael Wellman and Jon Doyle, “Preferential semantics for goals,” in Proceedings of the 9th National Conference on Artificial Intelligence (AAAI Press, 1991). This paper draws on a much earlier proposal by Georg von Wright, “The logic of preference reconsidered,” Theory and Decision 3 (1972): 140–67.
(21)
My late Berkeley colleague has the distinction of becoming an adjective. See Paul Grice, Studies in the Way of Words (Harvard University Press, 1989).
(22)
The original paper on direct stimulation of pleasure centers in the brain: James Olds and Peter Milner, “Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain,” Journal of Comparative and Physiological Psychology 47 (1954): 419–27.
(23)
Letting rats push the button: James Olds, “Self-stimulation of the brain; its use to study local effects of hunger, sex, and drugs,” Science 127 (1958): 315–24.
(24)
Letting humans push the button: Robert Heath, “Electrical self-stimulation of the brain in man,” American Journal of Psychiatry 120 (1963): 571–77.
(25)
A first mathematical treatment of wireheading, showing how it occurs in reinforcement learning agents: Mark Ring and Laurent Orseau, “Delusion, survival, and intelligent agents,” in Artificial General Intelligence: 4th International Conference, ed. Jurgen Schmidhuber, Kristinn Thorisson, and Moshe Looks (Springer, 2011). One possible solution to the wireheading problem: Tom Everitt and Marcus Hutter, “Avoiding wireheading with value reinforcement learning,” arXiv:1605.03143 (2016).
(26)
How it might be possible for an intelligence explosion to occur safely: Benja Fallenstein and Nate Soares, “Vingean reflection: Reliable reasoning for self-improving agents,” technical report 2015-2, Machine Intelligence Research Institute, 2015.
(27)
The difficulty agents face in reasoning about themselves and their successors: Benja Fallenstein and Nate Soares, “Problems of self-reference in self-improving space-time embedded intelligence,” in Artificial General Intelligence: 7th International Conference, ed. Ben Goertzel, Laurent Orseau, and Javier Snaider (Springer, 2014).
(28)
Showing why an agent might pursue an objective different from its true objective if its computational abilities are limited: Jonathan Sorg, Satinder Singh, and Richard Lewis, “Internal rewards mitigate agent boundedness,” in Proceedings of the 27th International Conference on Machine Learning, ed. Johannes Furnkranz and Thorsten Joachims (2010), icml.cc/Conferences/2010/papers/icml2010proceedings.zip.

الفصل التاسع: التعقيدات: البشر

(1)
Some have argued that biology and neuroscience are also directly relevant. See, for example, Gopal Sarma, Adam Safron, and Nick Hay, “Integrative biological simulation, neuropsychology, and AI safety,” arxiv.org/abs/1811.03493 (2018).
(2)
On the possibility of making computers liable for damages: Paulius Čerka, Jurgita Grigienė, and Gintarė Sirbikytė, “Liability for damages caused by artificial intelligence,” Computer Law and Security Review 31 (2015): 376–89.
(3)
For an excellent machine-oriented introduction to standard ethical theories and their implications for designing AI systems, see Wendell Wallach and Colin Allen, Moral Machines: Teaching Robots Right from Wrong (Oxford University Press, 2008).
(4)
The sourcebook for utilitarian thought: Jeremy Bentham, An Introduction to the Principles of Morals and Legislation (T. Payne & Son, 1789).
(5)
Mill’s elaboration of his tutor Bentham’s ideas was extraordinarily influential on liberal thought: John Stuart Mill, Utilitarianism (Parker, Son & Bourn, 1863).
(6)
The paper introducing preference utilitarianism and preference autonomy: John Harsanyi, “Morality and the theory of rational behavior,” Social Research 44 (1977): 623–56.
(7)
An argument for social aggregation via weighted sums of utilities when deciding on behalf of multiple individuals: John Harsanyi, “Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility,” Journal of Political Economy 63 (1955): 309–21.
(8)
A generalization of Harsanyi’s social aggregation theorem to the case of unequal prior beliefs: Andrew Critch, Nishant Desai, and Stuart Russell, “Negotiable reinforcement learning for Pareto optimal sequential decision-making,” in Advances in Neural Information Processing Systems 31, ed. Samy Bengio et al. (2018).
(9)
The sourcebook for ideal utilitarianism: G. E. Moore, Ethics (Williams & Norgate, 1912).
(10)
News article citing Stuart Armstrong’s colorful example of misguided utility maximization: Chris Matyszczyk, “Professor warns robots could keep us in coffins on heroin drips,” CNET, June 29, 2015.
(11)
Popper’s theory of negative utilitarianism (so named later by Smart): Karl Popper, The Open Society and Its Enemies (Routledge, 1945).
(12)
A refutation of negative utilitarianism: R. Ninian Smart, “Negative utilitarianism,” Mind 67 (1958): 542–43.
(13)
For a typical argument for risks arising from “end human suffering” commands, see “Why do we think AI will destroy us?,” Reddit, reddit.com/r/Futurology/comments/38fp6o/why_do_we_think_ai_will_destroy_us.
(14)
A good source for self-deluding incentives in AI: Ring and Orseau, “Delusion, survival, and intelligent agents.”
(15)
On the impossibility of interpersonal comparisons of utility: W. Stanley Jevons, The Theory of Political Economy (Macmillan, 1871).
(16)
The utility monster makes its appearance in Robert Nozick, Anarchy, State, and Utopia (Basic Books, 1974).
(17)
For example, we can fix immediate death to have a utility of 0 and a maximally happy life to have a utility of 1. See John Isbell, “Absolute games,” in Contributions to the Theory of Games, vol. 4, ed. Albert Tucker and R. Duncan Luce (Princeton University Press, 1959).
(18)
The oversimplified nature of Thanos’s population-halving policy is discussed by Tim Harford, “Thanos shows us how not to be an economist,” Financial Times, April 20, 2019. Even before the film debuted, defenders of Thanos began to congregate on the subreddit r/thanosdidnothingwrong/. In keeping with the subreddit’s motto, 350,000 of the 700,000 members were later purged.
(19)
On utilities for populations of different sizes: Henry Sidgwick, The Methods of Ethics (Macmillan, 1874).
(20)
The Repugnant Conclusion and other knotty problems of utilitarian thinking: Derek Parfit, Reasons and Persons (Oxford University Press, 1984).
(21)
For a concise summary of axiomatic approaches to population ethics, see Peter Eckersley, “Impossibility and uncertainty theorems in AI value alignment,” in Proceedings of the AAAI Workshop on Artificial Intelligence Safety, ed. Huáscar Espinoza et al. (2019).
(22)
Calculating the long-term carrying capacity of the Earth: Daniel O’Neill et al., “A good life for all within planetary boundaries,” Nature Sustainability 1 (2018): 88–95.
(23)
For an application of moral uncertainty to population ethics, see Hilary Greaves and Toby Ord, “Moral uncertainty about population axiology,” Journal of Ethics and Social Philosophy 12 (2017): 135–67. A more comprehensive analysis is provided by Will MacAskill, Krister Bykvist, and Toby Ord, Moral Uncertainty (Oxford University Press, forthcoming).
(24)
Quotation showing that Smith was not so obsessed with selfishness as is commonly imagined: Adam Smith, The Theory of Moral Sentiments (Andrew Millar; Alexander Kincaid and J. Bell, 1759).
(25)
For an introduction to the economics of altruism, see Serge-Christophe Kolm and Jean Ythier, eds., Handbook of the Economics of Giving, Altruism and Reciprocity, 2 vols. (North-Holland, 2006).
(26)
On charity as selfish: James Andreoni, “Impure altruism and donations to public goods: A theory of warm-glow giving,” Economic Journal 100 (1990): 464–77.
(27)
For those who like equations: let Alice’s intrinsic well-being be measured by wA and Bob’s by wB. Then the utilities for Alice and Bob are defined as follows:
Some authors suggest that Alice cares about Bob’s overall utility UB rather than just his intrinsic well-being wB, but this leads to a kind of circularity in that Alice’s utility depends on Bob’s utility which depends on Alice’s utility; sometimes stable solutions can be found but the underlying model can be questioned. See, for example, Hajime Hori, “Nonpaternalistic altruism and functional interdependence of social preferences,” Social Choice and Welfare 32 (2009): 59–77.
(28)
Models in which each individual’s utility is a linear combination of everyone’s wellbeing are just one possibility. Much more general models are possible — for example, models in which some individuals prefer to avoid severe inequalities in the distribution of well-being, even at the expense of reducing the total, while other individuals would really prefer that no one have preferences about inequality at all. Thus, the overall approach I am proposing accommodates multiple moral theories held by individuals; at the same time, it doesn’t insist that any one of those moral theories is correct or should have much sway over outcomes for those who hold a different theory. I am indebted to Toby Ord for pointing out this feature of the approach.
(29)
Arguments of this type have been made against policies designed to ensure equality of outcome, notably by the American legal philosopher Ronald Dworkin. See, for example, Ronald Dworkin, “What is equality? Part 1: Equality of welfare,” Philosophy and Public Affairs 10 (1981): 185–246. I am indebted to Iason Gabriel for this reference.
(30)
Malice in the form of revenge-based punishment for transgressions is certainly a common tendency. Although it plays a social role in keeping members of a community in line, it can be replaced by an equally effective policy driven by deterrence and prevention — that is, weighing the intrinsic harm done when punishing the transgressor against the benefits to the larger society.
(31)
Let EAB and PAB be Alice’s coefficients of envy and pride respectively, and assume that they apply to the difference in well-being. Then a (somewhat oversimplified) formula for Alice’s utility could be the following:
Thus, if Alice has positive pride and envy coefficients, they act on Bob’s welfare exactly like sadism and malice coefficients: Alice is happier if Bob’s welfare is lowered, all other things being equal. In reality, pride and envy typically apply not to differences in well-being but to differences in visible aspects thereof, such as status and possessions. Bob’s hard toil in acquiring his possessions (which lowers his overall well-being) may not be visible to Alice. This can lead to the self-defeating behaviors that go under the heading of “keeping up with the Joneses.”
(32)
On the sociology of conspicuous consumption: Thorstein Veblen, The Theory of the Leisure Class: An Economic Study of Institutions (Macmillan, 1899).
(33)
Fred Hirsch, The Social Limits to Growth (Routledge & Kegan Paul, 1977).
(34)
I am indebted to Ziyad Marar for pointing me to social identity theory and its importance in understanding human motivation and behavior. See, for example, Dominic Abrams and Michael Hogg, eds., Social Identity Theory: Constructive and Critical Advances (Springer, 1990). For a much briefer summary of the main ideas, see Ziyad Marar, “Social identity,” in This Idea Is Brilliant: Lost, Overlooked, and Underappreciated Scientific Concepts Everyone Should Know, ed. John Brockman (Harper Perennial, 2018).
(35)
Here, I am not suggesting that we necessarily need a detailed understanding of the neural implementation of cognition; what is needed is a model at the “software” level of how preferences, both explicit and implicit, generate behavior. Such a model would need to incorporate what is known about the reward system.
(36)
Ralph Adolphs and David Anderson, The Neuroscience of Emotion: A New Synthesis (Princeton University Press, 2018).
(37)
See, for example, Rosalind Picard, Affective Computing, 2nd ed. (MIT Press, 1998).
(38)
Waxing lyrical on the delights of the durian: Alfred Russel Wallace, The Malay Archipelago: The Land of the Orang-Utan, and the Bird of Paradise (Macmillan, 1869).
(39)
A less rosy view of the durian: Alan Davidson, The Oxford Companion to Food (Oxford University Press, 1999). Buildings have been evacuated and planes turned around in mid-flight because of the durian’s overpowering odor.
(40)
I discovered after writing this chapter that the durian was used for exactly the same philosophical purpose by Laurie Paul, Transformative Experience (Oxford University Press, 2014). Paul suggests that uncertainty about one’s own preferences presents fatal problems for decision theory, a view contradicted by Richard Pettigrew, Transformative experience and decision theory, Philosophy and Phenomenological Research 91 (2015): 766–74. Neither author refers to the early work of Harsanyi, Games with incomplete information, Parts I–III, or Cyert and de Groot, Adaptive utility.
(41)
An initial paper on helping humans who don’t know their own preferences and are learning about them: Lawrence Chan et al., “The assistive multi-armed bandit,” in Proceedings of the 14th ACM/IEEE International Conference on Human–Robot Interaction (HRI), ed. David Sirkin et al. (IEEE, 2019).
(42)
Eliezer Yudkowsky, in Coherent Extrapolated Volition (Singularity Institute, 2004), lumps all these aspects, as well as plain inconsistency, under the heading of muddle—a term that has not, unfortunately, caught on.
(43)
On the two selves who evaluate experiences: Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus & Giroux, 2011).
(44)
Edgeworth’s hedonimeter, an imaginary device for measuring happiness moment to moment: Francis Edgeworth, Mathematical Psychics: An Essay on the Application of Mathematics to the Moral Sciences (Kegan Paul, 1881).
(45)
A standard text on sequential decisions under uncertainty: Martin Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, 1994).
(46)
On axiomatic assumptions that justify additive representations of utility over time: Tjalling Koopmans, “Representation of preference orderings over time,” in Decision and Organization, ed. C. Bartlett McGuire, Roy Radner, and Kenneth Arrow (North-Holland, 1972).
(47)
The 2019 humans (who might, in 2099, be long dead or might just be the earlier selves of 2099 humans) might wish to build the machines in a way that respects the 2019 preferences of the 2019 humans rather than pandering to the undoubtedly shallow and ill-considered preferences of humans in 2099. This would be like drawing up a constitution that disallows any amendments. If the 2099 humans, after suitable deliberation, decide they wish to override the preferences built in by the 2019 humans, it seems reasonable that they should be able to do so. After all, it is they and their descendants who have to live with the consequences.
(48)
I am indebted to Wendell Wallach for this observation.
(49)
An early paper dealing with changes in preferences over time: John Harsanyi, “Welfare economics of variable tastes,” Review of Economic Studies 21 (1953): 204–13. A more recent (and somewhat technical) survey is provided by Franz Dietrich and Christian List, “Where do preferences come from?,” International Journal of Game Theory 42 (2013): 613–37. See also Laurie Paul, Transformative Experience (Oxford University Press, 2014), and Richard Pettigrew, “Choosing for Changing Selves,” philpapers.org/archive/PETCFC.pdf.
(50)
For a rational analysis of irrationality, see Jon Elster, Ulysses and the Sirens: Studies in Rationality and Irrationality (Cambridge University Press, 1979).
(51)
For promising ideas on cognitive prostheses for humans, see Falk Lieder, “Beyond bounded rationality: Reverse-engineering and enhancing human intelligence” (PhD thesis, University of California, Berkeley, 2018).

الفصل العاشر: هل حُلَّت المشكلة؟

(1)
On the application of assistance games to driving: Dorsa Sadigh et al., “Planning for cars that coordinate with people,” Autonomous Robots 42 (2018): 1405–26.
(2)
Apple is, curiously, absent from this list. It does have an AI research group and is ramping up rapidly. Its traditional culture of secrecy means that its impact in the marketplace of ideas is quite limited so far.
(3)
Max Tegmark, interview, Do You Trust This Computer?, directed by Chris Paine, written by Mark Monroe (2018).
(4)
On estimating the impact of cybercrime: “Cybercrime cost $600 billion and targets banks first,” Security Magazine, February 21, 2018.

الملحق «أ»: البحث عن حلول

(1)
The basic plan for chess programs of the next sixty years: Claude Shannon, “Programming a computer for playing chess,” Philosophical Magazine, 7th ser., 41 (1950): 256–75. Shannon’s proposal drew on a centuries-long tradition of evaluating chess positions by adding up piece values; see, for example, Pietro Carrera, Il gioco degli scacchi (Giovanni de Rossi, 1617).
(2)
A report describing Samuel’s heroic research on an early reinforcement learning algorithm for checkers: Arthur Samuel, “Some studies in machine learning using the game of checkers,” IBM Journal of Research and Development 3 (1959): 210–29.
(3)
The concept of rational metareasoning and its application to search and game playing emerged from the thesis research of my student Eric Wefald, who died tragically in a car accident before he could write up his work; the following appeared posthumously: Stuart Russell and Eric Wefald, Do the Right Thing: Studies in Limited Rationality (MIT Press, 1991). See also Eric Horvitz, “Rational metareasoning and compilation for optimizing decisions under bounded resources,” in Computational Intelligence, II: Proceedings of the International Symposium, ed. Francesco Gardin and Giancarlo Mauri (North-Holland, 1990); and Stuart Russell and Eric Wefald, “On optimal game-tree search using rational meta-reasoning,” in Proceedings of the 11th International Joint Conference on Artificial Intelligence, ed. Natesa Sridharan (Morgan Kaufmann, 1989).
(4)
Perhaps the first paper showing how hierarchical organization reduces the combinatorial complexity of planning: Herbert Simon, “The architecture of complexity,” Proceedings of the American Philosophical Society 106 (1962): 467–82.
(5)
The canonical reference for hierarchical planning is Earl Sacerdoti, “Planning in a hierarchy of abstraction spaces,” Artificial Intelligence 5 (1974): 115–35. See also Austin Tate, “Generating project networks,” in Proceedings of the 5th International Joint Conference on Artificial Intelligence, ed. Raj Reddy (Morgan Kaufmann, 1977).
(6)
A formal definition of what high-level actions do: Bhaskara Marthi, Stuart Russell, and Jason Wolfe, “Angelic semantics for high-level actions,” in Proceedings of the 17th International Conference on Automated Planning and Scheduling, ed. Mark Boddy, Maria Fox, and Sylvie Thiebaux (AAAI Press, 2007).

الملحق «ب»: المعرفة والمنطق

(1)
This example is unlikely to be from Aristotle, but may have originated with Sextus Empiricus, who lived probably in the second or third century CE.
(2)
The first algorithm for theorem-proving in first-order logic worked by reducing firstorder sentences to (very large numbers of) propositional sentences: Martin Davis and Hilary Putnam, “A computing procedure for quantification theory,” Journal of the ACM 7 (1960): 201–15.
(3)
An improved algorithm for propositional inference: Martin Davis, George Logemann, and Donald Loveland, “A machine program for theorem-proving,” Communications of the ACM 5 (1962): 394–97.
(4)
The satisfiability problem — deciding whether a collection of sentences is true in some world — is NP-complete. The reasoning problem — deciding whether a sentence follows from the known sentences — is co-NP-complete, a class that is thought to be harder than NP-complete problems.
(5)
There are two exceptions to this rule: no repetition (a stone may not be played that returns the board to a situation that existed previously) and no suicide (a stone may not be placed such that it would immediately be captured — for example, if it is already surrounded).
(6)
The work that introduced first-order logic as we understand it today (Begriffsschrift means “concept writing”): Gottlob Frege, Begriffsschrift, eine der arithmetischen nachgebildete Formelsprache des reinen Denkens (Halle, 1879). Frege’s notation for first-order logic was so bizarre and unwieldy that it was soon replaced by the notation introduced by Giuseppe Peano, which remains in common use today.
(7)
A summary of Japan’s bid for supremacy through knowledge-based systems: Edward Feigenbaum and Pamela McCorduck, The Fifth Generation: Artificial Intelligence and Japan’s Computer Challenge to the World (Addison-Wesley, 1983).
(8)
The US efforts included the Strategic Computing Initiative and the formation of the Microelectronics and Computer Technology Corporation (MCC). See Alex Roland and Philip Shiman, Strategic Computing: DARPA and the Quest for Machine Intelligence, 1983–1993 (MIT Press, 2002).
(9)
A history of Britain’s response to the re-emergence of AI in the 1980s: Brian Oakley and Kenneth Owen, Alvey: Britain’s Strategic Computing Initiative (MIT Press, 1990).
(10)
The origin of the term GOFAI: John Haugeland, Artificial Intelligence: The Very Idea (MIT Press, 1985).
(11)
Interview with Demis Hassabis on the future of AI and deep learning: Nick Heath, “Google DeepMind founder Demis Hassabis: Three truths about AI,” TechRepublic, September 24, 2018.

الملحق «ﺟ»: عدم اليقين والاحتمال

(1)
Pearl’s work was recognized by the Turing Award in 2011.
(2)
Bayes nets in more detail: Every node in the network is annotated with the probability of each possible value, given each possible combination of values for the node’s parents (that is, those nodes that point to it). For example, the probability that Doubles12 has value true is 1٫0 when D1 and D2 have the same value, and 0٫0 otherwise. A possible world is an assignment of values to all the nodes. The probability of such a world is the product of the appropriate probabilities from each of the nodes.
(3)
A compendium of applications of Bayes nets: Olivier Pourret, Patrick Naïm, and Bruce Marcot, eds., Bayesian Networks: A Practical Guide to Applications (Wiley, 2008).
(4)
The basic paper on probabilistic programming: Daphne Koller, David McAllester, and Avi Pfeffer, “Effective Bayesian inference for stochastic programs,” in Proceedings of the 14th National Conference on Artificial Intelligence (AAAI Press, 1997). For many additional references, see probabilistic-programming.org.
(5)
Using probabilistic programs to model human concept learning: Brenden Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum, “Human-level concept learning through probabilistic program induction,” Science 350 (2015): 1332–38.
(6)
For a detailed description of the seismic monitoring application and associated probability model, see Nimar Arora, Stuart Russell, and Erik Sudderth, “NET-VISA: Network processing vertically integrated seismic analysis,” Bulletin of the Seismological Society of America 103 (2013): 709–29.
(7)
News article describing one of the first serious self-driving car crashes: Ryan Randazzo, “Who was at fault in self-driving Uber crash? Accounts in Tempe police report disagree,” Republic (azcentral.com), March 29, 2017.

الملحق «د»: التعلم من التجربة

(1)
The foundational discussion of inductive learning: David Hume, Philosophical Essays Concerning Human Understanding (A. Millar, 1748).
(2)
Leslie Valiant, “A theory of the learnable,” Communications of the ACM 27 (1984): 1134–42. See also Vladimir Vapnik, Statistical Learning Theory (Wiley, 1998). Valiant’s approach concentrated on computational complexity, Vapnik’s on statistical analysis of the learning capacity of various classes of hypotheses, but both shared a common theoretical core connecting data and predictive accuracy.
(3)
For example, to learn the difference between the “situational superko” and “natural situational superko” rules, the learning algorithm would have to try repeating a board position that it had created previously by a pass rather than by playing a stone. The results would be different in different countries.
(4)
For a description of the ImageNet competition, see Olga Russakovsky et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision 115 (2015): 211–52.
(5)
The first demonstration of deep networks for vision: Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25, ed. Fernando Pereira et al. (2012).
(6)
The difficulty of distinguishing over one hundred breeds of dogs: Andrej Karpathy, “What I learned from competing against a ConvNet on ImageNet,” Andrej Karpathy Blog, September 2, 2014.
(7)
Blog post on inceptionism research at Google: Alexander Mordvintsev, Christopher Olah, and Mike Tyka, “Inceptionism: Going deeper into neural networks,” Google AI Blog, June 17, 2015. The idea seems to have originated with J. P. Lewis, “Creation by refinement: A creativity paradigm for gradient descent learning networks,” in Proceedings of the IEEE International Conference on Neural Networks (IEEE, 1988).
(8)
News article on Geoff Hinton having second thoughts about deep networks: Steve LeVine, “Artificial intelligence pioneer says we need to start over,” Axios, September 15, 2017.
(9)
A catalog of shortcomings of deep learning: Gary Marcus, “Deep learning: A critical appraisal,” arXiv:1801.00631 (2018).
(10)
A popular textbook on deep learning, with a frank assessment of its weaknesses: François Chollet, Deep Learning with Python (Manning Publications, 2017).
(11)
An explanation of explanation-based learning: Thomas Dietterich, “Learning at the knowledge level,” Machine Learning 1 (1986): 287–315.
(12)
A superficially quite different explanation of explanation-based learning: John Laird, Paul Rosenbloom, and Allen Newell, “Chunking in Soar: The anatomy of a general learning mechanism,” Machine Learning 1 (1986): 11–46.
الرجوع إلى الصفحة الرئيسية للكتاب

جميع الحقوق محفوظة لمؤسسة هنداوي © ٢٠٢٤