Devlog

The catalog's notable people grow from 13,094 to 23,034

A Wikipedia backfill adds 9,792 notable people and lifts covered groups from 292 to 453. Three days later, 15,228 rows carry a photo link.

  • Catalog
  • Data

On August 7 the notable-people backfill completed. It read Wikipedia's lists of notable people for the catalog's groups and saved 9,792 new people, taking the public groups with at least one notable person from 292 to 453. The phenotype catalog dataset shipped as v4 the same day: 23,034 notable people across 499 groups, up from 13,094.

On August 10 a second pass looked up each remaining person's Wikipedia page for a photograph. Photo links went from 5,883 rows to 15,228, about two thirds of the table.

Notable people give each group page real names, dates and faces. They are also the portrait source for the site's vision work: every per-image phenotype observation in the open dataset is a reading of one of these photographs.

Under the hood

  • Source pages are each group's "List of X people" article. A title that resolves to anything other than a real list page, such as a redirect to the main article about the people, is skipped rather than harvested.
  • Every entry passes a person-shape check: no event, place, ethnonym or year-leading names, and the list item must open with the linked person. The guards live in one shared module, so the scraper and the checks that replay them against stored rows apply identical rules.
  • Descriptions are reduced to a short known_for value with life-date parentheticals stripped, and birth and death years are stored as their own fields.
  • Photo enrichment is fill-in: it writes an image link where none exists and never overwrites one, so it reruns safely. Requests are paced per group and back off whenever Wikipedia asks them to.
  • Each row carries ethnic_id, name, known_for, birth and death years, a reference URL and an image URL, and joins to the per-image observations through example_id.
people added Aug 7
9,792
notable people in v4
23,034
rows with a photo link
15,228

Peoples in this entry

See it on the site

Sources and open data

Read the full log