All projects

N9/10AI & agentsResearchComplexity: Involved

PixelProof

A factory of verified visual puzzles that tests whether AI vision models really read gauges, charts and diagrams.

Status
Complete
Year
2026
Built for
Vision-language model evaluation
Languages
Python
Team
Sujal Negi
01

Overview

PixelProof generates, validates and scores visual-reasoning tasks from real operational settings: reading industrial gauges, business charts and wiring diagrams, where one misread value leads to a wrong decision.

Every answer is re-derived from the pixels by an independent verifier, and every task ships with a “counterfactual twin” that differs in one decisive detail. Same question, different correct answer, so a model that guesses from the text alone scores zero.

02

Problem and aim

Problem

Many visual benchmarks can be partly solved without looking at the image, and some answers in them are simply wrong. That makes it hard to know what a vision model can really see.

Aim

A reusable, reproducible pipeline that proves every question needs the image and every answer is correct.

03

How it works

01 / 06

Sample and render

A scene is sampled from one of six families (gauge, chart, wires, count, spatial, maze) at a chosen difficulty, and drawn.

02 / 06

G1 · Evidence

The image must carry enough evidence: readable tick spacing, clear bar gaps, wide wire crossings, no hidden objects.

03 / 06

G2 · Verifier

An independent algorithm recomputes the answer from the pixels. If it disagrees with the generator, the task is rejected.

04 / 06

G3 · Twin

One minimal edit (a needle, a bar, two swapped wire ends) must change the answer while the question stays identical.

05 / 06

G4 · Dedup

A perceptual-hash check removes near-duplicate images.

06 / 06

Evaluate

Models are scored strictly: no multiple choice, explicit tolerances, pair accuracy and an error taxonomy, in a self-contained HTML report.

04

Architecture

A generator, a gauntlet and an evaluator

AGenerate

  • Familiesgauge, chart, wires, count, spatial, maze.
  • RendererAnti-aliased canvas from a seed.

BValidate (G1–G4)

  • EvidenceIs the answer visible?
  • Pixel verifierRecompute the answer from pixels.
  • TwinOne edit must change the answer.
  • DedupPerceptual hash.

CEvaluate

  • Model adaptersGroq, OpenAI-compatible, Gemini, Anthropic, Ollama.
  • Strict scorerTolerances, pair accuracy, error types.
  • Report + StudioHTML findings and a local web UI.

Request flow

  1. pixelproof generate samples scenes, renders them and runs each through four gates.
  2. Accepted tasks go to tasks.jsonl with images and a manifest of the seed and rejections.
  3. pixelproof eval queries a model over HTTPS, with caching and a call budget.
  4. pixelproof report writes a self-contained HTML report; frontier finds breaking levels.
05

Languages and code

  • Python100.0%

Repository

31 files4,155 linesTap any file to read its code

  • data/2 files
    • demo/A verified demo dataset, evaluation results and the report.
  • deck/2 files
  • demo/2 files
    • 76 lines
    • 164 lines
  • pixelproof/The Python package.
    • families/One file per task family, each with sample, render, question, answer, twin and verify.
      • 2 lines
      • 237 lines
      • 155 lines
      • 204 lines
      • 176 lines
      • 202 lines
      • 233 lines
    • 3 lines
    • 5 lines
    • HTML report, local web UI and the command line. 124 lines
    • Data model, canvas, shape classifier and the family registry. 382 lines
    • Strict parsing, cached evaluation, metrics and breaking-point search. 55 lines
    • HTTP adapters for model providers and text-only controls. 236 lines
    • The validity gates, dataset writer and audit. 161 lines
    • HTML report, local web UI and the command line. 260 lines
    • Strict parsing, cached evaluation, metrics and breaking-point search. 186 lines
    • HTML report, local web UI and the command line. 332 lines
  • tests/Verifier agreement, twin discrimination, determinism and scoring.
    • 71 lines
  • 8 lines
  • 12 lines
  • 16 lines
  • 18 lines
  • 164 lines
  • 26 lines
  • 24 lines
06

Methodology

  1. Generate tasks from a seed, so any dataset can be reproduced exactly.
  2. Validate every task through four gates before it is used.
  3. Run text-only controls to prove the image is required.
  4. Search adaptively for the difficulty level where each model breaks.
07

Technology

CorePythonGenerator, verifiers, scorer and CLI.
NumPy + PillowRendering and pixel analysis.
EvaluationModel adaptersGroq, OpenAI-compatible, Gemini, Anthropic, Ollama and more.
PixelProof StudioLocal web UI to generate twins and query a model live.
08

Demo

09

Results

FamilyTasks correctPair accuracy
Report chart reading20/20100%
Wire tracing14/1586%
Spatial relations14/2060%
Analog gauge reading9/2040%
Route planning1/200%
Conjunction counting0/200%
Overall58/11546%
Qwen 3.8 27B (served on Groq), two twin pairs per family and level
Text-only controls scored 0% pair accuracy, confirming that every question needs the image.
10

Concepts

01What is a vision-language model?

An AI model that takes an image and text together and answers in text, for example “what does this gauge read?”.

02What is a counterfactual twin?

A second version of the same task with one decisive change, so the correct answer changes but the question does not. Getting both right is strong evidence the model really looked.

03Why “pair accuracy” as the headline?

A model can get one twin right by luck or by guessing a common value. Requiring both twins to be right removes most of that luck.

04What is a text-only control?

The same model, or a smart guesser, given the question without the image. If it scores well, the benchmark is leaking answers through the text.

11

Team