Devlog

The authoring engine: 10,837 entities, one rubric, 253 new pages

A Wikidata harvest, a written rubric and a four-step pipeline published 253 new peoples pages in one day, from the Mosuo to the Hidatsa to the Tzotzil.

  • Catalog
  • AI
  • Data

We rebuilt the candidate pool of the world's peoples from a harvest of 10,837 Wikidata entities typed as ethnic groups, and compared every one against the existing catalog. What survives the harvest goes through a written rubric before a word of the page is authored.

The rubric decides which peoples get a page: no governments, no census buckets, no uncontacted peoples, and no name that serves as a slur. Wave 1 published 253 new pages on September 6, from the Mosuo to the Hidatsa to the Tzotzil.

Wave 1 also brought historical peoples into the catalog as a cohort of their own: 80 pages, among them the Normans, the Anglo-Saxons, the Olmecs, the Xiongnu and the Varangians. The catalog now reaches back in time as well as across the map.

Under the hood

  • The harvest feeds four stages: review sheet, rubric review, authoring and publish. Candidates run in order of Wikidata sitelink count, so the best-documented peoples publish first, and every page enters the catalog hidden and goes public with a single switch.
  • Duplicate detection normalizes names after stripping a trailing group noun, so "Zulu people" meets "Zulu". A containment check flags pairs such as Baloch and Balochi, or San and Khoisan-San, for review instead of rejecting them, because the same test also fires on distinct neighbors such as Han and Hani.
  • The rubric has nine classes, tested in a set order. Every candidate leaves with exactly one outcome and one reason code, and the review sheet shows its Wikidata ID, its sitelink count, the opening sentence of its Wikipedia article and any live pages it resembles.
  • A language model reads the Wikipedia lead and authors each page: a description of 351 to 457 words in wave 1, plus structured fields for homeland, homeland region from a 23-region vocabulary, language, religion, subgroup, phenotype profile and portrait style register. A page is written when it passes validation, and nothing is invented to fill a gap.
  • Many historical peoples, such as the Normans and the Xiongnu, carry no era word in their names, and Wikidata marks few of them: 22 entities in the whole candidate pool carry a dissolution or end date. The rubric's test is whether any living community identifies by the name today, which places the Beothuk and the Batavi in the historical cohort and keeps the Hopi and the Karelians among the living.
Wikidata entities harvested
10,837
pages published in one day
253
historical peoples
80

Peoples in this entry

See it on the site

Sources and open data

Read the full log