What is computer use?
Reach for it when a vendor's own portal is the only door left, and the login screen changes six months later without telling you.
Computer use
computer-using agentGUI agent
Computer use is an AI capability where a model looks at a screenshot, decides where to move the cursor, and clicks or types. It works the way a person operates software, so an agent can act on a system with no API.
There is no structured data going in. The model gets a picture, works out what it is looking at, and picks an action: move here, click, type this string. Then it takes a new screenshot and does it again.
That loop is what lets it operate a legacy desktop app, a supplier portal with no API, or a government site built for a mouse. It is also slower and pricier than calling an endpoint. A layout change can break it overnight.
It is a last resort, and a useful one
We size the approach to what the target system actually offers. An API gives you a stable contract: named fields, versioned changes, an error you can catch. Build there first.
No API, but a predictable web flow? Browser automation that reads the page's structure, not its pixels, is next. It is faster and cheaper than computer use, and it survives more small changes.
Computer use is what is left when a vendor gave you nothing else. A desktop application, a portal that blocks scripted browsers, a screen you can only reach the way a person does. It is the right tool there, not everywhere.
- ScreenshotThe model sees the current screen.
- InterpretWhat is on it, and what changed.
- DecideMove, click, type, or scroll.
- ActThe action runs on the real screen.
- VerifyNew screenshot confirms it landed.
Every step reads pixels, not data. That is the whole cost difference against an API call.
Three ways to reach a system
Same goal, three different amounts of ground truth to work with.
| Criterion | API integration | Computer use |
|---|---|---|
| What the agent sees | Typed fields, with a schema. | Pixels. It has to interpret the screen itself. |
| Speed per action | One network call. | A screenshot, a model call, then the action. |
| What breaks it | A breaking change in the API version. | A moved button or a redesigned page. |
| Error handling | A status code you can catch. | Has to notice the screen looks wrong. |
| When we use it | Whenever the system exposes one. | Only when nothing else reaches the system. |
Related questions
01Is computer use the same as RPA?
No. Traditional RPA replays a recorded sequence of clicks at fixed screen coordinates, so it breaks the moment anything moves. Computer use interprets the screen at each step, which lets it adapt to small layout changes RPA cannot.
02Why not automate the browser instead?
Browser automation reads the page's underlying structure, which is faster and more reliable than reading pixels. Computer use is for the cases browser automation cannot reach: desktop software, or a portal that blocks scripted browser access outright.
03Is computer use reliable enough for production?
For narrow, well-tested tasks with a human reviewing the outcome, yes. Anything irreversible and unattended, such as a payment or a record deletion, needs a person approving that specific step.
04Does computer use need a specialised model?
It needs a model trained to interpret screenshots and output precise coordinates, which is a different skill from ordinary text generation. Not every model is built for it, so check that capability before you design a workflow around it.
