- AI
- Research
- Data
The generator comparison started as a way to check portraits. It is now a benchmark: a fixed set of peoples, the same prompt per people, rendered through six generators several times each, with every render scored by the same vision analysis the catalog uses and shown to readers for a verdict.
The first phase covers 82 peoples that have real-photo baselines, so a render's likeness can be measured against something. The second phase renders the peoples that have no portrait at all, and the best render of each becomes that people's first portrait, marked on its page as a benchmark render.
Each generator has a page with its renders, its scores and where it goes wrong, and the methodology page says how the scores are made and what the readers' verdicts add.
Under the hood
- A render plan is a grid of peoples by generators by repeats. The runner walks the grid in bounded calls, records every attempt with its outcome, and skips a cell that already has its renders, so a run can stop and resume without repeating work.
- A render that stalls is cancelled after a fixed wait and recorded as failed rather than holding the run; a generator that refuses a prompt is recorded with the refusal and never asked again for that cell.
- Scores come from the catalog's vision analysis against the people's phenotype profile, dimension by dimension; a validity study on renders with known baselines reports how well the score tracks the baseline.
- The reader deck pairs renders of the same people from two generators and asks which looks more like the people; the pair order and the generator identity are hidden until after the vote.
- Every render, score and verdict is written to the open dataset, so the benchmark can be rebuilt from its rows.
- generators compared
- 6
- baseline peoples
- 82
- of scored renders
- thousands
See it on the site
Related build notes
- Aug 31, 2026Measuring the ruler: 73 of 84 dimensions pass a test-retest studyA test-retest study over 196 portraits scored the head and neck dimensions, wrote the results into the vocabulary files, and made them citable.
- Aug 31, 2026A corpus of 28,756 facial and body observations over 39 peoples2,681 portraits read by a vision model against 22 controlled vocabularies now yield 28,756 structured observations that feed the trait pages.
- May 6, 202613,094 notable people and 5,668 photos read by a vision modelGroup pages gain Notable People linked to Wikipedia, and a vision model turns 5,668 reference photos into structured trait observations, each traceable.