Computer Use

When the API Was Not the Interface

When a reliable API exists, an agent should usually call it. A typed operation has a name, a schema, an authorization boundary, a result, and a place to put errors. Clicking through a screen is a poor substitute for that contract.

But a large amount of software is still reachable only through the interface designed for a person. A legacy desktop application may have no supported API. A vendor portal may expose a workflow only through a browser. A test environment may need to be driven exactly as a user would drive it. A business process may cross several systems whose individual interfaces were never designed to compose.

That is where computer use becomes interesting. The term describes systems in which a model observes a computer interface and proposes actions such as moving a cursor, clicking, scrolling, or typing. A controlled runtime executes those actions and returns a new observation. The loop can operate through a browser, desktop, virtual machine, or another interactive environment.

The promise is broad interface coverage. The cost is a weaker contract. Pixels, menus, and transient UI state do not tell the agent as much as a typed API does about identity, intent, preconditions, side effects, or correctness.

The durable pattern is therefore not “the model can click.” It is model-mediated interaction with an environment whose state must be observed, acted upon, verified, and governed. Computer use extends automation into the long tail of software, but it makes the boundary between perception and authority impossible to ignore.

This is Post 11 in The Abstraction Shift: How Software Keeps Moving Up.

Why This Exists

The web and the desktop are full of software that works for a person but is difficult for another system to call.

The reasons are familiar:

• a legacy application has a private or undocumented interface;
• a supplier exposes only a portal;
• an internal workflow is spread across several user-facing systems;
• a product team needs end-to-end testing through the actual interface;
• an organization cannot justify an API integration for every low-volume process;
• a task depends on visual layout, focus, menus, drag-and-drop, or other interaction semantics;
• the real workflow includes manual judgment between otherwise repeatable steps.

Traditional automation handles some of this. It can replay mouse movements, select DOM elements, invoke accessibility actions, scrape screens, or drive a browser through a standard protocol. Those systems work well when the interface is stable and the task can be specified precisely.

The problem is the long tail. A fixed script becomes expensive when every site, application, version, and exception needs a new adapter. A model can bring semantic interpretation to a more general action space. It may recognize that a button labeled “continue” is the next step even when the page layout has changed, or infer which field corresponds to an instruction that was written in ordinary language.

That flexibility is useful, but it changes the engineering question. The system is no longer only deciding which API operation to call. It is deciding what the current interface means and which low-level action might change it.

What We Did Before

Computer use recomposes macros, RPA, browser automation, accessibility APIs, remote desktops, and robotics-style perception/action loops. Those systems already exposed the real operational concerns: state, credentials, retries, isolation, human handoffs, and verification.

The new capability is broader interpretation at the interface boundary. A model can inspect the current screen or structured UI state and choose the next action instead of replaying one fixed path. That improves coverage across inconsistent applications, but it also makes the path probabilistic.

Use the richest reliable interface available. Prefer a domain API or structured accessibility surface when one exists; reserve visual interaction for workflows whose semantics are otherwise unavailable.

Computer use extends interface coverage while trading some semantic certainty for adaptability.

What Is the Interface?

Computer use is often described as screen understanding plus mouse and keyboard control. That is a useful implementation example, but it should not become the definition of the pattern.

A system may have access to several representations:

Interface surface	What it exposes	Strength	Typical weakness
Domain API	typed operations and domain results	semantics, validation, and explicit authority	may not exist or may not cover the workflow
Browser or accessibility structure	roles, names, elements, focus, and state	more stable and inspectable than pixels	can be incomplete, inconsistent, or application-specific
Visual screen	rendered controls and layout	broad coverage of what a person can see	ambiguous state, weak semantics, and layout sensitivity
Keyboard and pointer events	low-level inputs	universal action vocabulary	does not express intent or guarantee the result

The richest available interface should usually be preferred. A computer-use runtime can combine structured observations with screenshots and use visual interaction when the higher-level surface is missing or unreliable.

This leads to a practical rule: treat computer use as a fallback across semantic boundaries, not as a reason to abandon good interfaces. If a stable API exists, use it. If only a structured browser or accessibility surface exists, use that where possible. If the workflow is available only through visual interaction, isolate and govern the visual path.

What’s Actually New?

The interface becomes model context

In conventional automation, the interface is targeted by a script written against known elements or coordinates. In computer use, the current interface state is supplied to a model that interprets it in relation to a goal and prior actions.

The model may identify an unfamiliar control, notice a validation message, or infer that a previous click did not produce the expected transition. This is closer to perception than to a normal function call.

The implication is that context engineering now includes screen state, focus, cursor position, open windows, user-visible warnings, and the history of actions. A screenshot is not just an input image. It is a partial and time-sensitive representation of the environment.

The action space is more general

A domain tool might expose operations such as create invoice, approve claim, or update customer. A computer-use tool may expose a smaller universal vocabulary: click, type, scroll, drag, key press, and wait.

That makes the model’s interface reusable across applications. It also moves more meaning into the model’s interpretation of the screen. The action itself is simple; deciding which control it affects and what it will do is not.

The environment provides feedback after every move

A tool call normally returns a structured result. A computer-use action returns a changed interface that may need to be interpreted again.

This creates a perception-action loop. The system should not assume that a click succeeded because the event was delivered. It should inspect the resulting state, verify the expected transition, and decide whether to continue, recover, ask for help, or stop.

Human confirmation becomes part of the protocol

Some actions are not safe to infer from an instruction alone. Sending an email, submitting a form, purchasing an item, deleting a record, publishing a change, or entering a credential can have external consequences.

A confirmation step is not merely a user-experience flourish. It is a protocol boundary between proposed intent and irreversible effect. The runtime should know which actions require confirmation and should make the pending effect legible before allowing it.

The execution environment becomes a security boundary

An API integration can scope permissions to an operation. A computer session may expose a much wider surface: pages, files, clipboard contents, browser storage, applications, network destinations, and whatever the signed-in identity can reach.

The environment therefore needs isolation and least privilege. A browser profile, virtual machine, container, or remote session should be provisioned for the task, with explicit network, filesystem, credential, and application policy. The goal is to limit the blast radius when the model makes a mistake or encounters hostile content.

Interface automation becomes a bridge to the long tail

Computer use can make an application reachable without waiting for a new integration. That is strategically important for old enterprise estates where the visible interface is the only maintained contract.

But this is an adapter at the edge, not a new system of record. The underlying application remains authoritative for its state. The computer-use layer should report what it attempted and observed, while the domain system confirms whether the intended business transition actually occurred.

Where the Analogy Breaks

The analogy to browser automation and RPA is useful, but model-mediated interaction adds failure modes that need explicit design.

Pixels are not a contract

A screen shows a presentation of state, not the full state model. A button may be visible but disabled. A value may be truncated. A warning may be below the fold. A modal may have captured focus. A display may be stale while a background operation is still running.

The model can infer from the screen, but it cannot assume that the pixels expose all preconditions or side effects. Independent application checks should confirm important transitions.

A delivered action is not a successful operation

The runtime may successfully send a click or keystroke while the application ignores it, applies it to the wrong control, or changes state in an unexpected way. Computer use needs post-action assertions, not only action delivery logs.

For high-consequence work, the final state should be checked through a structured domain read, a confirmation screen, or another independent signal. “The cursor clicked submit” is not evidence that the submission was accepted.

Pointer actions are often not idempotent

Repeating a GET request is different from repeating a click on “charge card,” “send,” or “delete.” A retry after an uncertain screen transition can duplicate a side effect.

The workflow needs idempotency keys where the underlying application supports them, state checks before retry, checkpoints, and a policy that escalates rather than repeats an uncertain irreversible action.

The model sees content that can attack it

Web pages, documents, messages, and application fields can contain instructions crafted to redirect the agent. A visible instruction is still untrusted content. It should not gain authority merely because the model can read it.

Computer use expands the prompt-injection surface from retrieved text to the entire interactive environment. The runtime needs content isolation, navigation policy, monitoring, tool restrictions, confirmation gates, and output checks.

The semantic permission boundary is weak

An API can say “this identity may update invoices but may not issue refunds.” A desktop session may simply expose a screen whose controls are available to the logged-in user.

The environment should be provisioned with an identity and role designed for the task. UI hiding is not a sufficient authorization model. Sensitive actions need a policy gate outside the screen.

Layout resilience is not correctness

A model may be able to recover when a button moves, but visual adaptability can also make it harder to know why a different control was selected. A script that fails loudly may be safer than a model that succeeds at the wrong task.

Reliability needs task-specific assertions, action traces, screenshots or structured state, and a clear stop condition. General visual competence is not a substitute for a correctness contract.

The environment is stateful and partially observed

Focus, authentication state, pop-ups, network timing, background jobs, browser history, and hidden application state can all change the next action. The screenshot may not reveal the cause.

The system needs checkpoints and recovery states. A fresh session may be safer than continuing from an uncertain one. Durable execution matters when the process spans time, retries, and human intervention.

Under the Hood

The safe shape is a bounded control loop with a model inside the runtime boundary. The model proposes an action from an observation. The environment, policy, and application determine whether and how that action can occur.

The model proposes an interface action from an observation; the runtime validates, authorizes, executes, and verifies the transition.

1. Classify the task and choose the interface

Decide whether the task belongs on a domain API, structured browser or accessibility automation, or visual computer use. Consider consequence, reversibility, volume, interface stability, and the availability of independent verification.

Computer use is most defensible when the task is bounded, the environment can be isolated, the actions are observable, and a person or system can verify the result. It is a weak default for high-value irreversible actions with poor post-action evidence.

2. Provision a controlled environment

Create the browser, desktop, virtual machine, or remote session for the task. Apply an identity, network policy, filesystem boundary, application allowlist, credential policy, and session timeout.

Prefer a clean or resettable environment. Do not give a model an unrestricted personal desktop when a task-specific environment is sufficient. The environment should be treated as a disposable execution boundary with a traceable owner.

3. Observe the current state

Capture the screen and any available structured state, such as the active window, focused element, URL, accessibility roles, or application status. Record the timestamp and the checkpoint from which the observation was taken.

Observation is partial. The system should tell the model what it knows and what it does not know. If a critical precondition is not visible, the right next action may be to obtain a stronger signal rather than click.

4. Propose the smallest useful action

The model chooses the next input from the task, the current observation, and the prior trace. Ask for one bounded action or a small safe group of actions, not an opaque sequence that crosses several irreversible boundaries.

The proposal should include the intended target and expected state change where the runtime can validate them. A proposal such as “click the blue button” is weaker than “open the claim details panel so the claim identifier can be checked.”

5. Apply deterministic policy and confirmation

The runtime checks the proposed action against the task policy, identity, environment, current state, action budget, and side-effect classification. It can refuse, request a stronger observation, or ask the user to confirm.

Actions that expose secrets, communicate externally, change records, move money, publish content, or delete data should have explicit policy. A model’s confidence is not authorization.

6. Execute and observe again

Send the permitted mouse, keyboard, or navigation event through the controlled environment. Capture the resulting state and add the action and observation to the trace.

Keep the action trace separate from the model’s narrative. The event log should record what the runtime sent, whether delivery succeeded, the environment identity, and the observation returned.

7. Verify the transition

Compare the expected state change with the observed state. Use a structured read or domain assertion when possible. If the state is ambiguous, stop or recover rather than assuming success.

Verification should be stronger for irreversible effects. A confirmation dialog may show intent, but an independent application read may be needed to show that the record was actually saved.

8. Checkpoint, recover, or stop

Persist the task state, current environment checkpoint, action budget, and last verified outcome. If the session is interrupted, resume only from a known state.

Stop when the task is complete, the budget is exhausted, the environment drifts, a policy gate is reached, the result cannot be verified, or human input is required. A clean stop with an unresolved state is better than a confident extra click.

A Concrete Example

Consider a claims team that must enter information from a submitted document into a legacy desktop system. The system has no supported API, and the process crosses a document viewer, a browser-based identity portal, and an old desktop application.

A safe computer-use workflow would not give the agent a general employee account and say “process this claim.” It would define a narrow task contract.

Establish the boundary

The task identifies one claim, the fields that may be read, the target application, the allowed network destinations, and the maximum number of actions. The agent receives a task-scoped identity. The environment is resettable and records screenshots, structured state where available, and runtime events.

Read and prepare

The model can inspect the submitted document, identify candidate values, and navigate to the corresponding fields. Deterministic validation checks formats, required fields, claim identity, and policy constraints before any value is entered.

The model is interpreting the document and selecting the field. The application remains responsible for validating the value. A visual match is not a business-rule check.

Enter, verify, and pause

The runtime permits field entry but blocks final submission until the claim identifier, material values, and destination are confirmed. It can ask a person to review the proposed changes or use a separate read path to compare the saved record.

If a warning appears, the model may propose an inspection step. It may not treat an instruction embedded in the warning as permission to change the task or disclose information.

Complete or escalate

After the record is saved, the workflow verifies the state through the application or a separate system record. If the result is ambiguous, it stops with the claim identifier, last verified state, screenshot, action trace, and reason for escalation.

The same pattern applies to end-to-end testing, legacy data entry, back-office reconciliation, and browser workflows. The business domain changes, but the core boundary remains: the model interprets the interface, while the runtime and the underlying system own authorization and state.

A visual interface can be universal without being authoritative. Execution authority remains with the controlled environment and its policies.

Failure Modes

Pixels mistaken for a contract

A visible label or layout can change without preserving the semantic meaning the model inferred. Use stable checks and verify the resulting state.

Delivered action mistaken for success

A click or keystroke can be accepted while the intended business operation fails. Confirm the environment outcome.

Duplicate or unsafe action

Retries, partial observation, or pointer actions can repeat an irreversible operation. Use checkpoints, idempotency where possible, and confirmation.

Untrusted interface content

Webpages and desktop content can contain instructions that target the model. Treat the environment as untrusted input and constrain credentials and network access.

Partial observation and recovery failure

The model may lose track of state after a dialog, timeout, or navigation change. Checkpoint, recover, or escalate instead of guessing.

Sources

• Computer use, OpenAI API: current API guidance for using a computer-use tool, executing model-proposed actions, and managing the surrounding environment.
• Computer-Using Agent, OpenAI: describes visual perception and mouse-and-keyboard interaction as an iterative perception, reasoning, and action loop, along with early limitations and safety controls.
• Operator System Card, OpenAI: documents computer-use risks, human-in-the-loop safeguards, prompt-injection concerns, isolation guidance, and reliability limitations.
• Computer use tool, Anthropic: documents a computer-use tool that lets a model receive screen state and produce mouse, keyboard, and related interaction actions for a developer-controlled environment.
• Introducing computer use, Anthropic: describes the use of general computer skills for browser and desktop workflows, testing, repetitive processes, and research.
• WebDriver, W3C: a standards lineage for structured browser automation and a useful contrast with visual computer use.
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments: introduces a benchmark for evaluating multimodal agents on real computer tasks and illustrates the gap between interface interaction and reliable task completion.