Computer use: acting on the computer environment
Computer use is the layer where an agent acts on a computer — UI, desktop, browser, or OS-level actions. It is a form of tool use where the tool is the computer itself. This page explains what computer use is, why it raises the stakes for knowledge and guardrails, and how it fits in the agents stack.
What is computer use in an agentic system?
Computer use is the layer where an agent acts on a computer environment — clicking, typing, navigating UIs, operating desktop or browser applications, and performing OS-level tasks. It is a form of tool use where the tool is the computer itself.
Computer use as action at the UI level
Some actions cannot be done through a clean API. Sometimes the target is a UI, a desktop app, a browser, or an environment with no good programmatic interface.
Computer use is how an agent acts in those environments: navigating screens, reading UI state, clicking, typing, and operating within the computer as a human would. It is one of the most powerful forms of tool use because it can reach places that APIs do not expose cleanly.
That power is also what makes it risky. A computer-using agent can act on the user's environment directly, which means knowledge, permissions, and guardrails matter more than in a narrow API call.
Why computer use depends on knowledge
- UI state is knowledge. What is on screen, what is enabled, what changed — that is knowledge the agent must read and act on.
- Procedures are knowledge. Many computer tasks follow procedures: which app to open, which menu to use, what sequence of actions is correct. That is procedural knowledge.
- Permissions are knowledge. A computer-using agent often operates with significant access. It needs to know what it is allowed to do, on whose behalf, and under what constraints.
- Errors are expensive. A wrong click or action in a computer environment can be more consequential than a wrong sentence. Knowledge and guardrails reduce that risk.
In other words, computer use is not just an interface problem. It is a knowledge problem at the action layer.
Where computer use fits in the stack
In the agents stack on this site, computer use is the action layer at the UI/computer level:
- LLMs — the reasoning core.
- Agents — the loop that reasons, decides, and continues.
- MCP — one standard interface for tools and context.
- Tool use — the action layer generally.
- Computer use — tool use at the computer/UI level.
Computer use is valuable because it extends what an agent can do. It is risky because it expands the blast radius of bad knowledge and weak guardrails. Good computer-use systems pair action with strong knowledge and clear constraints.
Where to go next
Frequently asked questions
Computer use is the layer where an agent acts on a computer environment — clicking, typing, navigating UIs, operating desktop or browser applications, and performing OS-level tasks. It is a form of tool use where the tool is the computer itself.
Because computer use can act directly on the user's environment — files, applications, records, UI states — often with broad access. That makes knowledge, permissions, and guardrails especially important: the agent is no longer only generating text, it is operating in a real environment.
Computer use is a form of tool use at the UI/computer level. The stack is: LLMs reason, agents loop and decide, tools and interfaces like MCP expose capabilities, tool use is the action layer, and computer use is the action layer when the target is the computer environment.
Sources and further reading
- Retrieval-Augmented Generation for Large Language Models: A Survey — arxiv.org/abs/2312.10997