Category: artificial-intelligence
Type: argument
Associated thinker(s): Nick Bostrom
Original publication or approximate period: Bostrom, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents” (Minds and Machines, 2012)
Difficulty: Intermediate
Themes: goals, intelligence, AI safety
Related experiments: The Paperclip Maximizer; The AI Box Scenario
Nick Bostrom defines intelligence, for this argument, roughly as the capacity for instrumental reasoning: planning and choosing means to achieve ends. The orthogonality thesis says that “more or less” any level of this kind of intelligence could in principle be combined with “more or less” any final goal—the ends an agent values for their own sake.
So a highly capable agent might pursue an unfamiliar goal, such as counting grains of sand or maximizing paperclips, rather than a human goal. The thesis is about possible combinations, not a claim that every goal would be equally easy to implement or would leave practical competence unchanged.
Is there a necessary connection between an agent’s level of instrumental intelligence and its final goals?
Intelligence alone does not tell us what a particular system ultimately values, so capability by itself is not evidence of benevolence. Goal formation, inheritance, and implementation need separate attention; Bostrom also notes that logical possibility does not make it practically easy to install an arbitrary or human-respecting goal.
Bostrom distinguishes the orthogonality thesis from instrumental convergence. Orthogonality concerns the range of possible final goals. Instrumental convergence concerns subgoals—such as acquiring resources—that may help agents with many different final goals in a wide range of situations. It does not say that every agent will pursue those subgoals in every situation.
Later authors dispute how far these theses support claims about existential risk. Müller and Cannon argue that a common route from the orthogonality thesis to AI existential risk may combine different senses of intelligence. Sharadin argues that the probabilistic and fit-based accounts of goal promotion he examines do not establish dangerous instrumental convergence regardless of an agent’s specific goal. These are published arguments, not settled refutations or consensus.
Bostrom’s thesis is not simply a Humean claim that reason can never affect ends. He argues that orthogonality can hold even if beliefs about objective moral facts could motivate an agent, provided the thesis is about instrumental intelligence rather than a broader form of normative rationality. A moral realist can question whether that is the right conception of intelligence, but this is a philosophical dispute, not a direct contradiction of Bostrom’s definition.
The thesis also has limits: Bostrom says “more or less,” notes constraints on how goals and cognitive capacities fit together, and distinguishes changes to an agent’s final goals from routine changes to its subgoals. Questions about whether a particular agent will retain or revise its goals over time are related, but they are not identical to the thesis.
The thesis does not say that all goals are equally likely in real systems, that any specific present-day AI has a particular goal, or that capability alone predicts what an agent will do. It denies a general necessary link between instrumental intelligence and final goals while allowing constraints on particular combinations. It is a claim about possibility, not a prediction about current AI.
Bostrom uses paperclip maximization as an illustration; he also discusses counting grains of sand and calculating pi. An agent’s goals may change, but that raises a separate question about goal stability and prediction; it does not by itself refute the orthogonality thesis. How difficult it is to implement a stable, human-respecting goal is likewise a practical engineering question, not something the thesis settles.
Bostrom’s author-hosted PDF was retrieved and relevant sections of the complete text were checked, including his definition of instrumental intelligence and qualifications (Section 1.2), the distinction between the two theses and goal-change discussion (Section 2), and his conclusion. Müller and Cannon’s 2022 peer-reviewed article and Sharadin’s 2025 peer-reviewed article were consulted for later arguments; neither is presented as a consensus position. This is an educational paraphrase, not a substitute for the primary texts or expert review.
Research check: 2026-11-01. Additional comparisons are teaching prompts, not established historical variations.