Gridlings-Bench

1,100 constraint-logic puzzles across 11 rule families, every board machine-verified to have exactly one solution. Free, one JSON file, CC BY 4.0. Built for evaluating LLM reasoning: answers are exact-match scored against the verified solution, so no judge model, no rubric, and no partial-credit ambiguity.

⬇ bench-v1.json (250 KB)

Why another puzzle benchmark

• Verified, not scraped. Every board was generated with an exhaustive solver in the loop: exactly one solution exists. Nonogram boards pass a stronger bar — solvable by row/column constraint propagation alone, no search required.
• 11 families, one metric. Sudoku-style benchmarks measure one rule system; Gridlings-Bench spans Graeco-Latin grids with adjacency constraints, binary balance, two-star placement, Hamiltonian paths with ordered waypoints, inequality and visibility Latin squares, dot-constraint and sum-constraint variants, fill puzzles and picture logic. Aggregate accuracy is hard to reach by overfitting one trick.
• Unlimited fresh boards. The open generators produce arbitrarily many new verified boards from seeds — contamination is a regeneration away from being fixed. This set is drawn from the site's public free-play pools; the daily boards are excluded.
• Difficulty tags. 30 easy / 40 medium / 30 hard per family, using each game's own size and clue-count ramp.

Format

One JSON object. families maps each family to its rules text and answer format; puzzles is a flat list of {id, family, difficulty, puzzle, solution}. Present the family rules plus the puzzle object to the model; compare its answer to solution (exact match). Report per-family and overall accuracy.

FAQ

Is it free for commercial use? Yes — CC BY 4.0. Attribute "Gridlings-Bench, play.agiscorecard.com/bench".

How do I know the solutions are right? Each generator ships a validate mode that re-checks every published board against its own solver; the boards here are the same objects served to human players daily.

Can I get more/harder boards? Run the generators with your own seeds, or open an issue — generating a private held-out set for a lab is cheap. Contact: reply address on agiscorecard.com/about.

Humans can play every family daily: Gridlings · Nonogram · Star Battle · full archive.