Article
Levels of Autonomy for AI Agents Working Paper - arXiv
arxiv.org
Quoted on this wiki
Every place a page here uses this source, in the order the words come in it.
Abstract “We argue that an agent’s level of autonomy can be treated as a deliberate design decision, separate from its capability and operational environment.” 1Introduction
While autonomy can unlock innovative applications that amplify the benefits of AI, it can also do the same for AI’s negative consequences, raising significant concerns about AI risks. Scholars contend that it is simultaneously more important and more difficult to anticipate harms from autonomous AI, especially as accountability for AI actions becomes harder to trace [6]. Such harms may manifest immediately (e.g., scams [12] and leaking private information [23]) or more gradually (e.g., human deskilling and loss of critical thinking [8], societal disempowerment [22]). In Anthropic’s Responsible Scaling Policy framework [2], systems that show “low-level autonomous capabilities” are classified as ones that pose “significantly higher risk.” Indeed, researchers have even argued that fully autonomous agents should not be developed [29]. Despite these concerns, research and development of agents continue to focus on automating tasks, likely in part due to how the current evaluation paradigm of using benchmarks narrowly focuses on autonomous task completion accuracy [19]. “That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices.” We first introduce a five-level framework (Table 1) for agent autonomy. Key to our framework is its user-centered view. We define an AI agent’s autonomy as the extent to which it is designed to act without user involvement, and thus build our framework around five roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer. Importantly, while an agent’s capabilities may influence its autonomy [30] (or agency [29]), our framework views autonomy as a property that can also be designed independently of capability. For instance, a capable agent—one that performs well on evaluation benchmarks—can still act with a low level of autonomy if it is required to consult its user before taking each action. Recognizing autonomy as a double-edged sword that yields both exciting benefits and consequential risks, our framework is meant for agent developers and relevant stakeholders to make informed calibrations of agent autonomy based on target use cases and desired user experiences.
Abstract “In this work, we define five levels of escalating agent autonomy,” 1Introduction
Abstract “characterized by the roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer.” 1Introduction
The earliest grand visions for AI promised intelligent agents that can learn our preferences, autonomously perform tasks on our behalf, and transform the way we consume information [24, 26]. Today, rapid growth in the capabilities of frontier generative AI models—in particular, large language models (LLMs)—renders these visions more achievable than ever. These models, which have demonstrated sophisticated understanding of natural language and our visual world, can equip AI agents with impressive autonomous capabilities, such as proactively resolving GitHub issues [18, 44], shopping online [45], and generally using computers in a human-like manner [1, 32]. Excitement over autonomous agents11 1 Henceforth, our use of “agents” will refer to AI agents rather than human agents, unless explicitly stated otherwise. is pervasive—autonomy is often touted as a new paradigm shift in AI by companies, developers, and the press [29]. This enthusiasm has been reinforced by product releases such as OpenAI’s Operator [32], Cognition’s Devin [41], and Google’s Deep Research [11], accelerating agents’ inclusion into society. “Despite these concerns, research and development of agents continue to focus on automating tasks, likely in part due to how the current evaluation paradigm of using benchmarks narrowly focuses on autonomous task completion accuracy [19].” In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice?
The earliest grand visions for AI promised intelligent agents that can learn our preferences, autonomously perform tasks on our behalf, and transform the way we consume information [24, 26]. Today, rapid growth in the capabilities of frontier generative AI models—in particular, large language models (LLMs)—renders these visions more achievable than ever. These models, which have demonstrated sophisticated understanding of natural language and our visual world, can equip AI agents with impressive autonomous capabilities, such as proactively resolving GitHub issues [18, 44], shopping online [45], and generally using computers in a human-like manner [1, 32]. Excitement over autonomous agents11 1 Henceforth, our use of “agents” will refer to AI agents rather than human agents, unless explicitly stated otherwise. is pervasive—autonomy is often touted as a new paradigm shift in AI by companies, developers, and the press [29]. This enthusiasm has been reinforced by product releases such as OpenAI’s Operator [32], Cognition’s Devin [41], and Google’s Deep Research [11], accelerating agents’ inclusion into society. “Such harms may manifest immediately (e.g., scams [12] and leaking private information [23]) or more gradually (e.g., human deskilling and loss of critical thinking [8], societal disempowerment [22]).” In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice?
The earliest grand visions for AI promised intelligent agents that can learn our preferences, autonomously perform tasks on our behalf, and transform the way we consume information [24, 26]. Today, rapid growth in the capabilities of frontier generative AI models—in particular, large language models (LLMs)—renders these visions more achievable than ever. These models, which have demonstrated sophisticated understanding of natural language and our visual world, can equip AI agents with impressive autonomous capabilities, such as proactively resolving GitHub issues [18, 44], shopping online [45], and generally using computers in a human-like manner [1, 32]. Excitement over autonomous agents11 1 Henceforth, our use of “agents” will refer to AI agents rather than human agents, unless explicitly stated otherwise. is pervasive—autonomy is often touted as a new paradigm shift in AI by companies, developers, and the press [29]. This enthusiasm has been reinforced by product releases such as OpenAI’s Operator [32], Cognition’s Devin [41], and Google’s Deep Research [11], accelerating agents’ inclusion into society. “Despite these concerns, research and development of agents continue to focus on automating tasks, likely in part due to how the current evaluation paradigm of using benchmarks narrowly focuses on autonomous task completion accuracy [19].” In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice?
Autonomy. In robotics and human-robot interaction, autonomy is broadly interpreted as “the ability to operate without a human operator for a protracted period of time” [3, 20]. We build off this definition and use autonomy to refer to the extent to which an AI agent is designed to operate without user involvement. Per our earlier definition, “user” can be a human or another AI agent. We use “designed to operate” because, as we later argue, autonomy is a design decision accompanied by a set of model- and user interface-based constraints that developers can use to shape agent behavior. “Involvement” is multifaceted and includes a spectrum of actions from direct control to light supervision. The multifaceted nature of involvement forms the foundation of our autonomy framework we will introduce later. “An agent may have low agency if the set of tools it has access to is limited (e.g., it can only call a calculator API but cannot browse the web), thereby restricting its capacity to carry out an intentional action.” 3Five Levels of Autonomy for AI Agents
→The user’s request is not straightforward in the sense that it needs to be decomposed into smaller subtasks to be completed effectively. The agent leaves this planning process to the user. As the user works, the agent begins paying attention to the user’s activities in the environment. This way, the user remains in charge of the workflow, while the agent provides the user with contextual assistance when requested, or even proactively suggests ways it can help. For example, when the user opens a web browser and navigates to a search engine, the agent suggests search queries related to the use of generative AI in the United States. As the user reads relevant reports on the topic, they can request agent summaries in a low-friction way (such as with the click of a button or a keyboard shortcut). The user can also highlight certain concepts they want to more deeply investigate, and the agent searches the web for related literature and presents them to the user. Later on, the user downloads some relevant datasets and opens a code editor to transform and visualize the data. The agent ‘‘follows’’ the user to the code editor and suggests code autocompletions. “if the agent proactively suggests actions, it does do not execute them until they are approved by the user.” Given this, some open questions agent developers may wrestle the when developing effective L1 agents may include:
In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice? “We define an AI agent’s autonomy as the extent to which it is designed to act without user involvement” We then explore one potential application of our framework by proposing an agent governance mechanism which we call autonomy certificates. These certificates are issued by a third-party governing body to agent developers to communicate behavioral characteristics of an agent to other developers and agents. We highlight the utility of autonomy certificates for risk assessment, safety framework design, and multi-agent systems engineering, and discuss one practical implementation.
In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice? “our framework is meant for agent developers and relevant stakeholders to make informed calibrations of agent autonomy based on target use cases and desired user experiences.” We then explore one potential application of our framework by proposing an agent governance mechanism which we call autonomy certificates. These certificates are issued by a third-party governing body to agent developers to communicate behavioral characteristics of an agent to other developers and agents. We highlight the utility of autonomy certificates for risk assessment, safety framework design, and multi-agent systems engineering, and discuss one practical implementation.
3Five Levels of Autonomy for AI Agents “A capable agent (i.e., one that performs well on capability benchmarks) can be designed to behave only semi-autonomously to elicit and incorporate user feedback at regular intervals, while a not-so-capable agent can behave autonomously when tackling well-scoped and simple tasks.” As we walk through our framework, we use a running example featuring an agent with a pre-defined model, set of tools, and operational environment. We show how agent developers may vary the autonomy of the agent even when these factors for model capability and agency are held constant. We also use the same user request across all five levels for clearer comparison. Below are the specifications for our example agent.