Devlog

One people, one page: identity matching at catalog scale

Wave 2 adds 492 peoples pages, each matched to its own Wikipedia article first, and wave 3 shows 226 look-alike names are distinct peoples.

  • Catalog
  • Data

Wave 2 published 492 more peoples pages on September 12 and took the catalog past 2,400. Before publishing, each page was matched to its own Wikipedia article: one article, one page, whatever the spelling.

Wave 3, on September 16, published 226 pages from names that looked like duplicates of existing ones until a name-by-name review showed each was its own people: the Pontic Greeks, the Peranakan, the Miami, the Rashaida and more. Each had surfaced because its name contains an existing one; read in context, each named a people of its own.

The same day the pipeline learned to read beyond English. Peoples documented in French, Russian or Indonesian Wikipedia, with no English article, can now get English pages.

Under the hood

  • Identity by article: the review resolves every candidate to a Wikipedia article. Two names for one people resolve to one article, so normalizing the article title and matching it against the live catalog turns an identity question into a lookup. That is how Ngbaka meets Mbaka, a pair that string distance alone cannot tell apart from noise.
  • String matching runs alongside it: small edit distances, and a sound-folded consonant skeleton that folds doubled letters and interchangeable consonants such as c, k and q.
  • Wave 3 drew on 476 well-documented names, each containing the name of a live page inside its own. Read one by one, many were coincidences of spelling: Rashaida contains Haida, Peranakan contains Akan, Miami contains Ami. The Pontic Greeks passed as a distinct subgroup of the Greeks, whose page was already live.
  • Passing candidates then clear a five-way screen: the same Wikipedia title as a live page, a Wikipedia redirect, the title's slug already live, a normalized-name twin, and an exact slug.
  • Any-language grounding: when no English article exists, the review sheet falls back through a set list of Wikipedia editions (French, Spanish, Portuguese, Russian, Indonesian, German and on) and records the language and title. The rubric review judges that lead as it would an English one, and the author reads it in its language and writes in US English. At the time, 578 candidates in the remaining pool with no English page had one in a major edition.
pages in wave 2
492
pages in wave 3
226
peoples pages after wave 2
2,447

Peoples in this entry

See it on the site

Sources and open data

Read the full log