Protocol v1.0

How we test

A runoff is only fair if every tool gets the same shot. Here's the whole rulebook. The full prompt set and scoring files live in our public repo.

protocol v1.0 · at a glance
10
fixed prompts per category, word for word
×3
runs per prompt, all published
2
independent judges, a third on disagreement
1 in 5
outputs checked by a human before launch

The run

Six steps, the same for every tool, every month.

  1. Fixed prompts. 10 prompts per category, pasted word for word. Pixel sizes and audio lengths are part of the prompt.
  2. Default settings. Default model and settings, fresh project, no reference images, no trained styles, no seeds.
  3. Three runs per prompt. Separate generations. If a tool returns several variations, we keep the first one it shows.
  4. Everything recorded. Date, plan, model name, credits charged, generation time, file format and size, edit time, licence on that plan.
  5. Into a real game. Sprites go into a blank Godot scene at 1x and 3x; sounds go on a real jump, a real coin, a real loop.
  6. Scored, then written up. Numbers first, verdict second. All runs published.

The rubric

CriterionWhat we checkScale
Usability in a real gameDoes it drop in and work: right size, frames, loop, length, words?0-4
ConsistencySame character across frames, palette respected, tiles align, loops match; steady quality across the 3 runs0-4
Edit time neededTimed fixes to make it game-ready, capped at 30 min (0-2 min = 4, over 30 = 0)0-4 + minutes
Licence / commercial useWhat the terms say for that exact plan on test dayYes / Limited / No / Unclear
Cost per usable assetDollars spent on the category divided by usable outputsUS$
"Usable" means: usability 3+, consistency 2+, 15 minutes of edits or less, and a licence that lets you ship it. Personal-use-only outputs get scored but never count as usable.

Cost per usable asset

dollars per credit = plan price / credits included
spent = credits charged across all runs × dollars per credit
cost per usable asset = spent / number of usable outputs

Free tiers would always "win" at $0, so for free-tier runs we show the actual $0 and a paid-equivalent cost at the cheapest paid plan's rate. Rankings use the paid-equivalent. We also show the full monthly plan price, because that's the real entry ticket.

Who scores

  1. Scripts first. Pixel size, frame count, transparency, duration, sample rate, leading silence, loop clicks.
  2. Two independent judges. Two separate scoring passes on the same files; a 2-point disagreement triggers a third.
  3. Human spot-check. At least 1 in 5 outputs checked by a person before anything goes live.

Corrections

Tool makers and readers can flag factual errors at [email protected]. Fixes are dated and logged on the results page. Scores only change if we made a mistake, never because someone asked nicely.