Devlog

Reader verdicts become an open research dataset

One-tap verdicts now land in a public table: 173 judgments across 108 groups in v5, with the labels open under CC BY 4.0.

  • Community
  • Data
  • Research

On August 17 reader verdicts joined the phenotype catalog dataset on Hugging Face as a new table, reader_verdicts. Version 5 carried 173 verdicts across 108 groups. A tap on the site now becomes a row in an open corpus.

The table is a preference label set over the catalog's generated portraits: one question per portrait, looks right or looks off. Each row records the page where the judgment was made, because what a reader has on screen shapes a likeness call.

The rows, columns and labels are open under CC BY 4.0. Each row links the portrait it judges, which anyone may fetch to inspect or evaluate the labels, and the images themselves stay with the site.

Under the hood

  • Core columns: image_id, ethnic_id, verdict with its label (looks_right or looks_off), surface, rater_id, image_url, created_at, and image_generator_recorded, which marks the portraits whose generating model is stored as provenance metadata.
  • surface is written once, at creation, and never rewritten, so a re-tap from another page cannot relabel where the original judgment formed. At v5 the values were ethnic_page and country_carousel; region_page and cluster_page joined about a week later.
  • rater_id is a pseudonym, stable within a release so inter-rater agreement is computable, and regenerated for each release.
  • The export keeps verdicts on portraits from the site's own generated catalog and leaves out any that point at an externally hosted photograph, since a crowd judgment about an identifiable real person does not belong in an open dataset.
  • image_id is the stable identifier for joins. Portraits get regenerated, so an image URL is current at release while the id holds across releases.
verdicts in v5
173
groups judged
108

Peoples in this entry

Sources and open data

Read the full log