Skip to content

Look files up in an index of the upload history - #958

Merged
rtwojcik merged 1 commit into
mainfrom
feat/fingerprint-history-index
Nov 19, 2026
Merged

rtwojcik merged 1 commit into
mainfrom
feat/fingerprint-history-index

Conversation

@rtwojcik

@rtwojcik rtwojcik commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follows #957 (merged).

With #957, upload and check_upload_history read every record of the upload history at start and hold all their files in memory. Reviewing #957 showed that with 1.5M uploaded files this takes seconds to tens of seconds and hundreds of MB on every run. The Desktop Uploader runs check_upload_history on every file drop.

This PR looks files up in a SQLite index next to the records instead. The records stay the source of truth: the index is brought up to date with them when opened, reading only the records added, changed or deleted since.

Changes

  • history_index.py (new): open_uploaded_files() opens the index, fingerprints_v1/index_v1.sqlite3.
    • Up to date: in one transaction, the record files are listed and only the ones whose status changed are read (size, modification and change times, and file ID, which isn't read on Windows); deleted ones are dropped. A record that can't be read, is invalid (including values SQLite can't store) or is of an unfinished upload is indexed without files, so it isn't read or warned about again until it changes.
    • Lookups go by size, and only then by fingerprint, as before: a file whose size no uploaded file had is not read. Files uploaded during the run are kept in memory on top of the index; their record is indexed the next time.
    • Several processes: WAL mode, so lookups aren't blocked while another process updates the index; the write-ahead log is cut back to 16 MB after a checkpoint. Updates wait for each other (up to 30 s).
    • Failures: a corrupt index (not a database, malformed) is rebuilt from the records, and so is one left read-only (like after a run as another user) when the folder can be written. Any other failure, at open or in a lookup (locked past the timeout, read-only folder, I/O error), falls back to reading every record, as in Look files up in the upload history by size and fingerprint #957, and keeps the index.
    • Versions: the schema version is in the file name, so mapillary_tools versions with different tables (a Desktop Uploader and a command line install) each keep their own index instead of rebuilding each other's.
    • First build: with 1,000 records or more to index, and more than are indexed, the SQL indexes are made after the files are inserted, which is faster; it's logged once.
  • history.py: UploadedFilesLookup, the interface of both lookups; parse_uploaded_files_record(), shared by both, so that they agree on what is uploaded (a file size larger than SQLite stores makes a record invalid). Records are written whole: to a temporary file, then moved in place, so that a reader never sees a partial record and an interrupted write doesn't lose the previous one (also true on main before).
  • upload.py: upload and check_upload_history use the index and close it when done. upload validates its options before opening the history.
  • Docs: check_upload_history changes nothing but the index (README and command reference). This changes what the docs promised; it's intended, as the Desktop Uploader runs check_upload_history on every file drop, where the index saves the most.

Measurements

A synthetic history of 15,000 records × 100 files (1.5M files, 176 MB of records), looking up 200 new files:

Linux x86, NVMe Apple Silicon Mac max RSS
#957, reading every record, every run 6.6 s 30 s (114 s with a cold disk cache) ~280 MB
This PR, first run (builds the index) 9.8 s 35 s 64 MB
This PR, later runs 0.05 s 0.1–0.2 s 32 MB

The index file is 121 MB for 1.5M files.

Review

Before opening this PR, an independent review (another AI agent) checked parity with reading every record, multi-process stress (2 writers + 4 readers, final index equal to the records), interrupts (Ctrl-C, SIGKILL mid-build), rebuild and fallback. Its findings, and those of a later review (atomic record writes, change detection beyond modification time and size, an index left read-only), are fixed here: a lookup failing during a run now falls back instead of aborting the upload; out-of-range values make a record invalid instead of crashing; only corruption deletes the index; the index is closed when upload fails early; the write-ahead log size is capped. A second review of the fixes found no blockers. It also found that a record with an upload time out of range could make upload fail when skipping one of its files; that is fixed in #957.

Not changed: a read-only upload history folder falls back to reading every record (as in #957), and WAL doesn't work across hosts on a network filesystem, which the upload history, kept on this computer, isn't meant for.

Known limits

  • The index is built in one transaction. If a run is killed while building a large index (after it was corrupted, or for a future format), the next run starts over. The first build is small: records by fingerprint start with Look files up in the upload history by size and fingerprint #957.
  • On Windows, a record rewritten at the same size with its modification time set back by hand isn't noticed: file IDs aren't compared there, as reading them takes a system call per record. mapillary_tools itself never writes a record like that.
  • A read-only upload history folder reads every record, as in Look files up in the upload history by size and fingerprint #957. WAL doesn't work across hosts on a network filesystem, which the upload history, kept on this computer, isn't meant for.

Test plan

  • ruff check, ruff format --check, usort diff, mypy mapillary_tools tests/cli: clean.
  • pytest tests: 1717 passed, 96 skipped (macOS, Python 3.12, ExifTool 13.60).
  • New tests/unit/test_history_index.py (21 tests, also each alone and after the CLI tests):
    • finds what the records hold, as reading every record does, also on a randomized history (sizes, fingerprints, latest upload time, unfinished uploads);
    • reads only records added or changed since (also a rewrite of the same size), drops deleted ones;
    • bulk indexing, and not in bulk on a populated index; invalid and out-of-range records warned about once; files uploaded during the run found;
    • a corrupt index, one of another version or one left read-only rebuilt; a record rewritten at the same size with its modification time restored read again; records written whole; a folder in its place, a locked index or an I/O error: every record read, the index kept; a lookup failing after open: every record read for the rest of the run;
    • upload closes the index when it fails; no history or nothing uploaded yet: nothing created.
  • The upload and check_upload_history tests of Look files up in the upload history by size and fingerprint #957 run through the index.

@meta-cla meta-cla Bot added the cla signed label Nov 19, 2026
upload and check_upload_history read every record by fingerprint at start
and held all their files in memory. They now look files up in a SQLite
index next to the records, brought up to date when opened: only records
added, changed or deleted since are read. An index that can't be read is
rebuilt from the records; when it can't be used, every record is read as
before.
@rtwojcik
rtwojcik force-pushed the feat/fingerprint-history-index branch from f0ebda9 to 1a67e6e Compare November 19, 2026 13:25
@rtwojcik
rtwojcik changed the base branch from feat/fingerprint-upload-history to main November 19, 2026 13:25
@rtwojcik
rtwojcik merged commit 554d72f into main Nov 19, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant