At Warmwind, we build agents for office work. These agents do not only call APIs or read clean web pages. They look at applications, choose a target, click it, type into it, and move through workflows. You see this kind of work in Google Docs, Excel, Slack, Canva, SAP, Salesforce, and internal tools (Warmwind OS, Warmwind closed beta announcement).

But we do not stop there. We work on robotics control policies and continuous action spaces: multiple actions give smoother control than single executions at a time.

But pointer models still matter. An LLM can plan a step such as “click submit,” but a policy still has to convert that intent into a screen location before the agent can act. That grounding step is the pointer model, which is why it shows up in evaluation, small or offline deployments, fallbacks, and agent systems that already plan with an LLM but have no reliable way to locate the target on screen.

Pointerbench is for anyone building computer-use systems, whether they call a large VLM through an API, fine-tune a small pointer model, or use it as an evaluation layer for a continuous controller. In each case it measures the same thing: whether the model can ground an action on the screen. A strong score does not imply general computer-use competence, but it does show that the model handles spreadsheets, text, colors, boundaries, relative positions, and professional UI targets, which is close to the work many teams need to automate.

It is also a demanding test of visual understanding in general, not only of clicking. The sheet tasks make this concrete: current VLMs still misread which color fills a spreadsheet cell or where a column border sits, even though the questions are simple. A model that misses those has not understood the screen; it has learned where common controls tend to sit.

We have trained a small pointer model for this kind of work and will publish it in a later release.

Why another benchmark?

ScreenSpot and ScreenSpot-Pro did important work. ScreenSpot-Pro moved GUI grounding into professional software. It covers 23 applications, 5 work categories, and 3 operating systems. In the original setting, existing models reached only 18.9 percent (ScreenSpot-Pro overview). The paper also shows that small targets and dense interfaces remain hard (ScreenSpot-Pro OpenReview).

Scores have since climbed steeply. At the time of this release the top public entries on the ScreenSpot-Pro leaderboard reach about 80.9 percent (ScreenSpot-Pro leaderboard), up from 18.9 percent. Fast saturation of a single benchmark makes it hard to tell whether models learned general GUI grounding or fit the distribution of that one benchmark. We built Pointerbench in part to test that difference on a separate, out-of-distribution set of tasks.

That direction is useful, but it leaves a gap: the work people ask us to automate is not mostly “click a clean web button.” It is office work: spreadsheets, documents, email, text fields, internal tools, and professional desktop applications.

In these environments, the target is often not an object. It can be a gridline, a cell corner, a caret position, a colored status cell, or a spreadsheet position such as “three rows below A5.” Object detection does not fit these cases well. It requires labels for too many pixels, edges, and insertion points.

Agents usually work inside one surface

Benchmarks that cover Windows, macOS, and Linux often suggest that production agents drive a full desktop. In practice, they usually do not. Most agent work happens inside a controlled surface: a browser tab, one application window, a cropped screenshot, or a rendered document, with one task on that surface.

That also matches how companies wire automation today. Tools such as n8n, UiPath, Make, and Zapier run the workflow. The model handles a scoped step on a specific surface. If you test the whole operating system, you give too much weight to desktop chrome that the agent rarely touches. Pointerbench focuses on the repeated unit of work: find the right target inside the given surface. Pointerbench-Pro still records platform, but it scores the captured surface, not the whole desktop.

Annotation ambiguity

Existing GUI grounding benchmarks also contain ambiguous prompts. A human can see more than one valid click target, but the benchmark accepts only one. Sometimes the prompt does not say enough to identify a single target.

These cases are not just model errors. They are label problems: the benchmark measures how well the model guesses one hidden label rather than cursor skill. Very high scores on datasets with many such cases may partly reflect data leakage or benchmark memorization rather than grounding ability.

The screenshots below are taken from ScreenSpot and ScreenSpot-Pro; the coloured labels are our own annotations, not model output. In each case the green marker is the single target the benchmark accepts, and every Also valid marker is a different target that satisfies the same instruction but is still scored as wrong.

Color grounding is worse than it looks

Color carries real meaning in spreadsheets: status cells, warning rows, approval states, and project categories. Models still confuse nearby colors and often select an adjacent color instead of the requested one.

Pointerbench-Sheets includes 56 color tasks across cells, rows, columns, and bands. Most screens deliberately contain a nearby distractor color: the model must read the requested color, not just find the largest colored region.

If a model cannot follow color instructions, it fails many spreadsheet workflows. We report color cases separately so this weakness does not disappear inside an average.

What we are releasing

Pointerbench has three public GUI grounding benchmarks. Each record has one image, one instruction, target geometry in absolute pixels, and a binary evaluator. Point tasks use point-in-bbox scoring. Bbox tasks, currently used in Pointerbench-Text, use an asymmetric overlap rule: the ground truth must be almost fully covered (coverage at least 0.90) while the prediction stays reasonably tight around it (precision at least 0.70). Cutting off part of the target is penalised far more than wrapping it with some margin.

The benchmark is available now: GitHub repository and Hugging Face dataset.

First leaderboard

The first release will report each benchmark separately. A single average hides useful information: whether a model handles spreadsheets, text and carets, or professional app interfaces.

To add your model to the official benchmark leaderboard, contact Warmwind.

The table ranks models by the reported average across the three subbenchmarks. Cells are shaded by score.

At the top of the table, Claude Fable 5 leads with a 78.7 percent average (96.0 on Sheets, 58.4 on Text, 80.8 on Pro), followed by our Pointer 1.6 in agentic mode at 77.1 percent. The GPT-5.6 Sol variants reach 65.3 percent, alongside the Claude 4.x models between 61.8 and 69.3 percent. Text is the hardest subset for every model: even the strongest entries lose 20 to 40 points there compared to Sheets, because caret positions and tight bounding boxes demand character-level localization. No model is close to saturating the suite.

Two things explain the low end of the table. First, several models often returned no parseable coordinate, which counts as a miss. Out of 500 examples per subbenchmark, Grok Build 0.1 and MiMo v2.5 returned nothing usable on every example in all three subbenchmarks, Grok 4.3 on all 500 Pro and 468 Text examples, Gemini 3.1 Pro on 357 Text examples, and Gemini 3.5 Flash on 319 Pro examples. Second, the center-click baseline always clicks the middle of the image, so it marks the score floor for a model with no real grounding.

The two benchmarks also rank models differently, which is the reason we report Pointerbench next to ScreenSpot-Pro rather than as a replacement. Qwen3.7-Plus reaches about 79 percent on ScreenSpot-Pro, fourth on the public leaderboard, but 3.1 percent on Pointerbench, with 5.8 percent on Sheets, 0.6 percent on Text, and 3.0 percent on Pro. That places it among the weakest models in the table above rather than near the 78.7 percent at the top. Anthropic and GPT models are the main exception: they rank near the top of both. A high ScreenSpot-Pro score therefore does not imply general grounding on its own; a model can fit one benchmark’s distribution without transferring to a broader set of pointing tasks.

Why some strong ScreenSpot-Pro models fail here

The effect we lean on is a loss of output freedom. A model tuned to click GUI targets learns to place its prediction on the center of a salient control: a button, a menu item, an input field, an icon. Across many training examples this is reinforced until little probability mass is left for anything else.

Its attention then acts like a magnet, and the common GUI elements are iron: the prediction is pulled onto the nearest button, label, or field and cannot settle on white space, an edge, or a gap between characters. Coordinates outside the training distribution become effectively unreachable, whatever the instruction says. An instruction like “point to the empty area next to the button” fails, because the model has learned to predict the center of controls, not the space beside them.

Pointerbench includes these off-element targets on purpose. Tasks such as “point to the left border of the cell”, “place the caret between two letters”, and the edge, corner, and relative-offset categories all require a coordinate that is not the center of a familiar element, which is exactly what a collapsed output space cannot produce.

Inference Protocol

All screenshots are 1024×768. The scorer expects absolute pixel coordinates in that image frame, with the origin at the top-left. If you resize, crop, or use a zoom loop during inference, convert the final answer back to 1024×768 before scoring and report that choice with your result.

The GitHub repo includes a default prompt helper:

python eval.py --show-system-prompt

Recommended system prompt:

You are evaluating Pointerbench, a GUI grounding benchmark. You will receive one 1024x768 screenshot and one task instruction. Use absolute pixel coordinates with origin at the top-left of the image. Do not return normalized coordinates. Do not crop or resize the coordinate frame. For point tasks, return JSON like {"point": [x, y]}. For bounding-box tasks, return JSON like {"bbox": [x0, y0, x1, y1]}, where [x0, y0] is the top-left corner and [x1, y1] is the bottom-right corner.

You can edit the prompt for your model or inference stack, but keep the coordinate frame fixed so scores stay comparable.

Pointerbench-Sheets

Spreadsheets are common in office work. They are also hard for GUI grounding. A sheet has dense grids, repeated cells, row and column labels, scroll offsets, colors, resize handles, grid edges, cell corners, and relative references.

We make this benchmark synthetic on purpose. Many models have seen Excel-like screenshots during training. They can learn shortcuts: A1 sits near this pixel, row numbers start here, the first column is visible. Pointerbench-Sheets removes those shortcuts. The visible grid does not always start at A1. Rows and columns scroll. The style changes. Some examples remove app chrome completely. Edge, corner, color, and relative tasks test whether the model understands the sheet structure.

Example instructions:

• Choose cell E11.
• Click the row that is colored red.
• Resize column I from its right edge.
• Place the cursor on B37's bottom-right grid intersection.
• From G10, go 3 columns left, then 3 rows down.

Selected examples

Edgecol_resize_handlelibreofficeEN

Prompt

Point to the draggable divider for column F.

Expected output

point [473, 116]

Colorcell_colorgoogle_sheetsEN

Prompt

Point to the blue-filled cell.

Expected output

point [199, 432]

Headercol_headergoogle_sheetsEN

Prompt

Can you select column H?

Expected output

point [548, 119]

Cornercell_top_left_cornerexcel_whiteEN

Prompt

Point to the top-left vertex of I35.

Expected output

point [408, 350]

Relativecell_relative_offsetlibreofficeEN

Prompt

Move from D4: 1 column right; 3 rows down.

Expected output

point [547, 287]

Cellcell_refexcelNL

Prompt

Click the grid cell named A13.

Expected output

point [108, 215]

Pointerbench-Text

Text creates many small targets. A model must click words, characters, punctuation, line starts, blank lines, and caret positions between letters. Some rows ask for a full bounding box around a word, line, character, or paragraph. These targets are not normal objects. Synthetic text gives us exact glyph geometry and controlled cases.

A dedicated invoice block adds 70 bounding-box targets over real invoice anatomy: invoice number, dates, sender and recipient blocks, individual line items and the full line-item table, subtotal, tax and VAT ID, total, and bank details (IBAN/BIC). These render on their own invoice surface so each field is an unambiguous target, and the line-item boxes include the header row and background bands rather than clipping the visible table.

The answer format is mixed: 366 examples ask for a point and 134 ask for a bounding box. Point examples use point-in-bbox scoring. Bbox examples use an asymmetric overlap rule (coverage at least 0.90 and precision at least 0.70) rather than plain IoU, so missing part of the target costs much more than overshooting it.

The language mix is 250 English examples and 50 examples each in German, French, Spanish, Italian, and Dutch. All non-English examples use Latin script.

Example instructions:

• Mark the center of `loaf` before '17.99'.
• Grab the bounding area for "aloof" following «act».
• Box in the entire paragraph that opens with the last "In".
• Klik op "birželio".
• The typing position between the 't' and the "o" in 'cartons' right after '6'.

Selected examples

Bboxinvoice_numberinvoiceDE

Prompt

Return the bounding box of the invoice number.

Expected output

bbox [557, 124, 635, 136]

Bboxinvoice_line_iteminvoiceEN

Prompt

Draw a box around the line item row for 'Consulting services'.

Expected output

bbox [276, 290, 718, 300]

Bboxinvoice_totalinvoiceEN

Prompt

Return the bounding box of the total payable amount.

Expected output

bbox [679, 399, 751, 415]

Wordword_centerdocs_siteDE

Prompt

Center of the text the "Strommarkt" in bold.

Expected output

point [851, 321]

Bboxline_bboxdiff_viewerEN

Prompt

Give the bounding box of the whole line starting with `+ # validate`: top-left then bottom-right, one coordinate per line.

Expected output

bbox [53, 268, 245, 283]

Bboxparagraph_bboxpdf_viewerEN

Prompt

Locate the paragraph whose opening text is «With» in the phrase `tolerance. With 34-unit` and return the bbox.

Expected output

bbox [187, 332, 837, 389]

Pointerbench-Pro

Professional applications are dense and specialized. They often make small targets harder to read than consumer web apps. Pointerbench-Pro complements ScreenSpot-Pro with generated GUI targets across professional software. The public set keeps the generated target mix instead of forcing an icon and text quota. Icon rows explicitly ask for the icon itself, so a model cannot pass by clicking the text label sitting next to it.

The benchmark covers 100 applications. Each record includes app, app_slug, app_category, platform, and source_id. You can report results by app or by app category.

The categories include development, creative tools, scientific tools, consumer apps, CAD and engineering, productivity, enterprise, office, communication, operating system surfaces, analytics, CMS, smart home, and finance.

Example instructions:

• Find a packet. (Wireshark)
• Switch to Home. (Microsoft Word)
• Select the red-coat clip. (Adobe Premiere Pro)
• Open favorite symbols. (KiCad)

Selected examples

OtherAdobe Premiere ProcreativeWindows 11

Prompt

Select the red-coat clip.

Expected output

point [649, 538]

IconWiresharkdevelopmentWindows 10

Prompt

Select the icon that performs this action: Find a packet.

Expected output

point [204, 58]

IconDiscord desktopcommunicationWindows 11

Prompt

Click the visual icon for this command: Continue to the next step.

Expected output

point [628, 538]

IconKiCad 8 PCB Editorcad_engineeringWindows 10

Prompt

Use the icon itself to do this: Place a rectangular zone.

Expected output

point [848, 310]

IconMicrosoft TeamscommunicationWindows 11

Prompt

Target the icon for this command: Open the meeting chat panel.

Expected output

point [471, 69]

TextPyCharmdevelopmentmacOS Sonoma

Prompt

Confirm settings.

Expected output

point [716, 546]

Format and evaluation

Each benchmark ships as an imagefolder dataset:

data/test/metadata.jsonl
data/test/0000.png
data/test/0001.png
...

Each metadata row contains the fields needed for scoring:

{
"file_name": "0000.png",
"id": "pbs_0000",
"instruction": "Click cell E11.",
"bbox": [x0, y0, x1, y1],
"point": [x, y],
"answer_type": "point",
"eval": {"type": "point_in_bbox", "bbox": [x0, y0, x1, y1]},
"data_type": "cell",
"category": "cell_ref",
"image_size": [1024, 768]
}

Predictions use JSONL. Each row contains an id and either a point or a bbox, depending on answer_type. A point is correct when it lands inside the target bbox. A predicted bbox is correct under the asymmetric overlap rule: it must cover at least 0.90 of the ground truth while keeping precision at least 0.70. Thin targets such as gridlines, cell corners, and carets have small boxes on purpose.

Why use synthetic data?

For Sheets and Text, synthetic data gives us exact geometry. That matters for gridlines, corners, caret locations, glyphs, colors, and relative positions. It also tests whether the model learned the structure, not just a familiar screenshot. If a model memorized common Excel layouts, it knows where A1 usually appears. Pointerbench can start the visible sheet at H23, remove the app chrome, scroll rows away, or ask for a boundary instead of a cell.

How to use Pointerbench

1. Use python eval.py --show-system-prompt to print the recommended system prompt, or edit it for your inference stack.
2. Run a model on every image and instruction.
3. Parse the model output as an absolute pixel point or absolute pixel bbox, depending on the row.
4. Run python eval.py --predictions preds.jsonl.
5. Report each subbenchmark separately.

git clone https://github.com/warmwindOS/pointerbench.git
cd pointerbench
huggingface-cli download WarmwindOS/pointerbench \
--repo-type dataset \
--local-dir .
cd pointerbench-sheets
python eval.py --show-system-prompt
python eval.py --predictions preds.jsonl

Release links

• Pointerbench GitHub repository
• Pointerbench dataset on Hugging Face
• Pointerbench-Sheets files
• Pointerbench-Text files
• Pointerbench-Pro files
• Our pointer model and Warmwind Robot Computer Use are internal for now; we will publish them in a later release.

References

• Warmwind OS: Building the AI Operating System for Everyone
• Warmwind: Autonomous Cloud Employees Enter Closed Beta
• ScreenSpot-Pro overview on HuggingFace
• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
• ScreenSpot-Pro leaderboard (llm-stats.com)
• WebArena: A Realistic Web Environment for Building Autonomous Agents
• WebArena paper PDF

Maximilian Schilling is CEO and CTO of Warmwind AG.
© Warmwind AG.