Overview
PixelProof generates, validates and scores visual-reasoning tasks from real operational settings: reading industrial gauges, business charts and wiring diagrams, where one misread value leads to a wrong decision.
Every answer is re-derived from the pixels by an independent verifier, and every task ships with a “counterfactual twin” that differs in one decisive detail. Same question, different correct answer, so a model that guesses from the text alone scores zero.
Problem and aim
Problem
Many visual benchmarks can be partly solved without looking at the image, and some answers in them are simply wrong. That makes it hard to know what a vision model can really see.
Aim
A reusable, reproducible pipeline that proves every question needs the image and every answer is correct.
How it works
01 / 06
Sample and render
A scene is sampled from one of six families (gauge, chart, wires, count, spatial, maze) at a chosen difficulty, and drawn.
02 / 06
G1 · Evidence
The image must carry enough evidence: readable tick spacing, clear bar gaps, wide wire crossings, no hidden objects.
03 / 06
G2 · Verifier
An independent algorithm recomputes the answer from the pixels. If it disagrees with the generator, the task is rejected.
04 / 06
G3 · Twin
One minimal edit (a needle, a bar, two swapped wire ends) must change the answer while the question stays identical.
05 / 06
G4 · Dedup
A perceptual-hash check removes near-duplicate images.
06 / 06
Evaluate
Models are scored strictly: no multiple choice, explicit tolerances, pair accuracy and an error taxonomy, in a self-contained HTML report.
Architecture
AGenerate
- Familiesgauge, chart, wires, count, spatial, maze.
- RendererAnti-aliased canvas from a seed.
BValidate (G1–G4)
- EvidenceIs the answer visible?
- Pixel verifierRecompute the answer from pixels.
- TwinOne edit must change the answer.
- DedupPerceptual hash.
CEvaluate
- Model adaptersGroq, OpenAI-compatible, Gemini, Anthropic, Ollama.
- Strict scorerTolerances, pair accuracy, error types.
- Report + StudioHTML findings and a local web UI.
Request flow
pixelproof generatesamples scenes, renders them and runs each through four gates.- Accepted tasks go to
tasks.jsonlwith images and a manifest of the seed and rejections. pixelproof evalqueries a model over HTTPS, with caching and a call budget.pixelproof reportwrites a self-contained HTML report;frontierfinds breaking levels.
Languages and code
- Python100.0%
Repository
data/2 files
demo/A verified demo dataset, evaluation results and the report.
- 176 lines
- report.htmllarge file
deck/2 files
- 447 lines
- PixelProof_deck.htmllarge file
demo/2 files
- 76 lines
- 164 lines
pixelproof/The Python package.
families/One file per task family, each with sample, render, question, answer, twin and verify.
- 2 lines
- 237 lines
- 155 lines
- 204 lines
- 176 lines
- 202 lines
- 233 lines
- 3 lines
- 5 lines
- HTML report, local web UI and the command line. 124 lines
- Data model, canvas, shape classifier and the family registry. 382 lines
- Strict parsing, cached evaluation, metrics and breaking-point search. 55 lines
- HTTP adapters for model providers and text-only controls. 236 lines
- The validity gates, dataset writer and audit. 161 lines
- HTML report, local web UI and the command line. 260 lines
- Strict parsing, cached evaluation, metrics and breaking-point search. 186 lines
- HTML report, local web UI and the command line. 332 lines
tests/Verifier agreement, twin discrimination, determinism and scoring.
- 71 lines
- 8 lines
- 12 lines
- 16 lines
- 18 lines
- 164 lines
- 26 lines
- 24 lines
Methodology
- Generate tasks from a seed, so any dataset can be reproduced exactly.
- Validate every task through four gates before it is used.
- Run text-only controls to prove the image is required.
- Search adaptively for the difficulty level where each model breaks.
Technology
Demo
Results
| Family | Tasks correct | Pair accuracy |
|---|---|---|
| Report chart reading | 20/20 | 100% |
| Wire tracing | 14/15 | 86% |
| Spatial relations | 14/20 | 60% |
| Analog gauge reading | 9/20 | 40% |
| Route planning | 1/20 | 0% |
| Conjunction counting | 0/20 | 0% |
| Overall | 58/115 | 46% |
Text-only controls scored 0% pair accuracy, confirming that every question needs the image.
Concepts
01What is a vision-language model?
An AI model that takes an image and text together and answers in text, for example “what does this gauge read?”.
02What is a counterfactual twin?
A second version of the same task with one decisive change, so the correct answer changes but the question does not. Getting both right is strong evidence the model really looked.
03Why “pair accuracy” as the headline?
A model can get one twin right by luck or by guessing a common value. Requiring both twins to be right removes most of that luck.
04What is a text-only control?
The same model, or a smart guesser, given the question without the image. If it scores well, the benchmark is leaking answers through the text.
Team
- Sujal NegiCo-founder, Nexara9PortfolioLinkedIn