We Benchmarked 10 Vision Models on Clicking What They See

An AI QA agent makes one promise that everything else depends on: it clicks through your app the way a real user would. Not by replaying a stored selector, but by looking at the rendered screen, finding the element, and clicking inside it. Strip away the product around it and that promise reduces to a mechanical question: given a screenshot, can a vision model tell you what is on screen and return X/Y coordinates that actually land inside the thing you asked it to click?

We benchmarked 10 vision models on exactly that question, because our agent stakes every test run on the answer. The result surprised us in one specific way: understanding a screenshot and interacting with it turned out to be two different capabilities, and models that ace the first can completely fail the second.

View on X (formerly Twitter)

The setup

The task sounds trivial. Each model received a screenshot and two kinds of work: understanding questions ("what is visible on this screen?") and click tasks ("return the coordinates to click this element").

The benchmark used two real screenshots:

• an iOS Settings screen at 1206×2622 pixels
• a web dashboard at 1280×800 pixels

Each screen had 4 click targets and 4 understanding questions. A click passed only when the returned source-pixel coordinate landed inside the audited target box. There was no partial credit for being close, because a real UI does not give partial credit either: the tap either hits the toggle or it hits the row below it.

Every model received identical conditions:

• the same screenshots and tasks
• the same structured output schema
• the source dimensions of each image
• temperature 0
• a 120 second timeout

Latency measured the model request only, with no agent overhead mixed in. Cost is the market cost reported by Vercel AI Gateway. We measured four things: grounding accuracy, visual understanding, latency, and cost.

A click passed only when the returned source-pixel coordinate landed inside the box.

Models ace the reading test and fail the pointing test

Here is the finding that matters more than any ranking. All ten models answered 100% of the visual understanding questions correctly, on both screens, without exception. The click column tells a different story: it runs from 100% straight down to zero, with half the field landing 50% or fewer of the targets and two models missing every single one.

In other words, a model can tell you, accurately and in fluent detail, that the screen shows iOS Settings with Wi-Fi enabled and Bluetooth on, and then place its click for the Wi-Fi row somewhere in the middle of General. It read the screen correctly. It just cannot point at what it read. Semantic understanding and pixel-accurate grounding are different capabilities, and benchmarks that only test the first will tell you nothing about whether a model can drive a UI.

Every model read both screens perfectly. Pointing at what they read is where the field splits.

This distinction is easy to miss if you evaluate models the way most people do, by chatting with them about images. Every model in the field has been trained extensively to describe what it sees. Far fewer have been trained to output coordinates that survive contact with a real coordinate system, at real resolutions, on real product screens.

For a QA agent the distinction is everything. A wrong answer in a chat is a correction. A wrong click in a test run is a wrong action: the agent toggles the wrong setting, opens the wrong menu, or reports a checkout flow broken because it pressed the button next to the one that mattered. An agent that understands your app but cannot ground its clicks does not produce slightly worse tests. It produces flaky ones, and flakiness is the exact disease teams are trying to escape when they leave scripted suites behind. We wrote about where that flakiness comes from in scripted suites in why your E2E tests keep breaking; an AI agent with weak grounding would simply reintroduce it through a different door.

The results

In the screening round, only three models achieved 100% on both understanding and grounding:

• GPT-5.6 Terra
• GPT-5.6 Luna
• Claude Sonnet 5

GPT-5.5 came close at 93.8% overall, and the split behind that number is the whole story of this benchmark in miniature: its understanding was perfect, and it landed 87.5% of the clicks. The faster, cheaper tier repeated the pattern more dramatically. GPT-5.4 and Claude Haiku 4.5 turned in the fastest requests in the field at 5.8 seconds, understood both screens completely, and still localized only half of the click targets, with Gemini 3.7 Flash matching them at 50%. Meta's Muse Spark 1.1 landed 12.5%. And both Qwen models, Qwen3-VL 235B and Qwen3.7 Flash, scored 100% on understanding while missing every single click target. Speed is not what these models are missing. Grounding is.

Here is the complete screening field:

The full field. One benchmark run per model, single screenshot per scene, market cost as reported by Vercel AI Gateway.

Notice what the cost column does not do: it does not predict grounding. The most expensive run in the field, GPT-5.5 at $0.059430, missed clicks that GPT-5.6 Luna landed for $0.002135, and the cheapest model of all, Qwen3.7 Flash at $0.000247, missed every target. You cannot buy grounding by spending more, and you cannot assume its absence because you spent less.

We then ran the three finalists independently for 5 repetitions each. These runs were measured separately from the screening pass, so their latency figures are not directly comparable to the screening column above:

Two numbers in that table deserve translation into physical reality. The iOS screenshot renders at 3x scale, where Apple's minimum recommended tap target of 44 points is 132 pixels wide. Claude Sonnet 5's mean center error of 128 pixels spans nearly the full width of a minimum-size control, which means that on small targets its clicks live at the edge of failure even when they pass. Luna's 16 pixels is a fingertip's wobble: essentially every click lands deep inside the intended element, run after run.

A 44-point tap target is 132 pixels wide at 3x. Luna's mean miss is 16 of them; Sonnet 5's is 128.

Note that all three finalists scored 100% on this benchmark's targets. Center error is what tells you how much margin that 100% carries. A model that passes by 2 pixels and a model that passes by 100 pixels look identical on a leaderboard and behave very differently on the long tail of dense menus, small icons, and mobile layouts that make up real products.

On balance across accuracy, precision, and cost, GPT-5.6 Luna won the benchmark. GPT-5.6 Terra won raw speed.

Why a QA agent needs all four axes at once

We measured grounding, understanding, latency, and cost together because an agent that tests real products on every pull request needs all four, and they trade against each other.

Grounding accuracy is the trust budget. When there is no selector to fall back on, the click either lands or the test lies. This is also what makes selector-free testing hold up when your UI changes: the agent finds the button by looking at the rendered screen, so a redesign that would break a stored CSS path is just another screenshot to it, provided the model behind it can actually point.

Precision is what repeatability is made of. A test that clicks the same element in the same place on every run behaves the same way on every run. Center error across repetitions is a direct, measurable proxy for how much run-to-run variance the model itself injects into your suite.

Latency compounds. A single click that takes 3.3 seconds instead of 5.7 seconds looks like a rounding error. A 30-step flow, run on every pull request, multiplies that difference into minutes of PR feedback time, and PR feedback time is the whole point: the value of testing every pull request is knowing the change works before anyone approves the merge.

Cost decides testing frequency. Per-action model cost is the difference between running your critical flows on every PR and running them nightly, and bugs found nightly are bugs found after the author has moved on. The screening numbers show how much room there is to get this right: Luna delivered perfect grounding at $0.002135 for the benchmark workload, roughly a ninth of what Terra and Sonnet 5 cost for the same perfect score, and 28 times less than GPT-5.5 spent to miss one click in eight.

This is why we benchmark instead of defaulting to whichever frontier model tops the general leaderboards. The best model for describing a screenshot, the best model for grounding a click, and the best model per dollar at PR frequency are not guaranteed to be the same model, and this round of results says they currently are not.

If you are evaluating an AI QA tool, or building on vision models yourself

The benchmark suggests four questions worth asking of any system that claims an AI can operate your UI, including ours:

1. Was grounding measured separately from understanding? A demo where the model narrates a screenshot proves the cheap half of the capability. Ask for click accuracy numbers, measured against audited target boxes, with no partial credit.
2. Was it measured at source resolution? Coordinates that work on a downscaled 800px image can miss by a hundred pixels on the 1206×2622 screenshot a real iPhone produces. Mobile is where grounding goes to die, and mobile is where your users are.
3. Was it repeated? One run measures luck. Five repetitions with a mean center error measure a capability. Precision across runs is what separates a stable suite from a flaky one.
4. What happens when the click misses anyway? No model is at zero error, so the system around the model has to catch the residue. Every TesterArmy run ships with step screenshots, a screen recording, and a written report, so a misfire surfaces as reviewable evidence rather than a silent wrong result. You review what the agent did, not just what it claims.

No model is at zero error, so every run ships with the evidence to review what the agent actually did.

That last point is the honest conclusion of the whole exercise. Grounding is improving fast, the best models are already precise to within a fingertip, and none of that removes the need for evidence. The benchmark tells us which models deserve to sit behind the agent. The recording of every run is what lets you verify each result, so you know what is broken before your customers do.

If you want to see grounding in practice rather than in a table, point the agent at your own product: the first five test runs are free, they work on a live URL or a mobile app binary, and no source code is required. Describe a flow in plain English and watch where the clicks land.