The problem
Choosing a generative model is usually done on one of two numbers, and both are wrong. Price per image ignores that rejected images cost the same as accepted ones. Benchmark quality ignores that you are not buying quality, you are buying assets that survive review.
Tosskeep generates game art through a quality gate, so the denominator was sitting right there: divide total spend by the number of assets that passed. That is the number a studio actually pays.
The finding
The result that changed the decision was asymmetric, and I would not have predicted it. Post-processing lifts a cheap general model enormously and actively damages an expensive purpose-trained one.
Quality scores from the bake-off, on a 100-point scale. Prices are per generated image, before any rejection.
Post-processing closed a 28-point gap to two points. On the run that settled it, the cheap path scored 85 at $0.030 per image against 80 at $0.360 for the purpose-trained one: better output, roughly a twelfth of the price.
Measuring raw scores alone would have led straight to buying the expensive model, and applying the post-processing pipeline uniformly would have quietly degraded it. Both mistakes are the default behavior.
The judge has to be judged
Ranking on approval rate only works if the gate approving things is any good, so two pieces sit underneath the metric.
- An anchored evaluator. Absolute scoring from a language model is miscalibrated because the model has no idea what this particular user considers good. So it gets up to three images the user already approved, as calibration points, upscaled four times with nearest-neighbour: bilinear would smooth the pixel art it is being asked to judge.
- Meta-evaluation. Each evaluator can be scored against human approve and reject verdicts on accuracy, rank correlation, false approvals and false rejections. It is an interface on the evaluator type rather than a script, so a new judge is graded the same way.
The frontier refuses to pick
The bake-off output is a Pareto frontier: every configuration not beaten on both cost and quality survives, and nothing is crowned. That is deliberate. Where a studio sits between "cheapest acceptable" and "best available" is a decision about their game, not about the data, and a tool that hides that behind a single recommendation is making a business call it is not entitled to make.
What this is not. Cost per approved asset ranks bake-offs. It is not a runtime router: nothing consumes it at generation time, and model choice still comes from a config table that its own header calls interim. Feeding the metric back into selection is the difference between having measured something and having built something, and I have only done the first.
What I would do differently
- Run the meta-evaluation against real human verdicts. The loop is built and tested and has never been run on live data, which means the judge is still unjudged in practice. It is the highest-value hour of work available in the project and it needs a person, not a machine.
- Close the loop on selection. See the caveat above. The metric has been sitting one step short of doing the job it was designed for.
- Bake off earlier. The comparison ran after the pipeline was built around a model choice made on intuition. The intuition was wrong by a factor of twelve in price.
The architecture from this project became a production art pipeline at work, rebuilt there in two weeks. It went that fast because the expensive questions, which model, how to judge it, what to do on failure, had already been answered here with data.
← All case studies