Faster re-indexing, and indexing that looks after itself

A re-index should cost re-indexing. Until now, rebuilding a search index meant analysing everything again — every page of recognised text, every picture — whether or not anything about it had changed since the last run. It no longer does. Text and images that have not changed are not re-analysed, and unchanged curated metadata is carried over in the same way, for items and for the articles inside them. On a large archive, a broad re-index becomes something you schedule rather than something you plan a weekend around.

And it commits as it goes. Re-indexing writes incrementally rather than in one large pass at the end, which removes a long-standing bottleneck on big collections. A manual or bulk run is visible in the interface while it works, reporting readable progress rather than turning a handle.

The similarity index looks after itself. The background pass that makes items eligible for More like this tops itself up on its own and resumes after a restart. No administrator action is needed: a large archive converges quietly, and items imported moments ago gain their neighbours as it catches up.

It also repairs what has drifted. The same pass can re-check every item against its stored description and refresh only the ones that no longer agree with it — so neighbours that went stale after a person was renamed, or after two person records were merged, come right without anyone asking them to.

What that indexing is for is shown working on the discovery page.