Repository navigation
Look files up in an index of the upload history - #958
Merged
Merged
Conversation
upload and check_upload_history read every record by fingerprint at start and held all their files in memory. They now look files up in a SQLite index next to the records, brought up to date when opened: only records added, changed or deleted since are read. An index that can't be read is rebuilt from the records; when it can't be used, every record is read as before.
rtwojcik
force-pushed
the
feat/fingerprint-history-index
branch
from
November 19, 2026 13:25
f0ebda9 to
1a67e6e
Compare
rtwojcik
changed the base branch from
feat/fingerprint-upload-history
to
main
November 19, 2026 13:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follows #957 (merged).
With #957,
uploadandcheck_upload_historyread every record of the upload history at start and hold all their files in memory. Reviewing #957 showed that with 1.5M uploaded files this takes seconds to tens of seconds and hundreds of MB on every run. The Desktop Uploader runscheck_upload_historyon every file drop.This PR looks files up in a SQLite index next to the records instead. The records stay the source of truth: the index is brought up to date with them when opened, reading only the records added, changed or deleted since.
Changes
history_index.py(new):open_uploaded_files()opens the index,fingerprints_v1/index_v1.sqlite3.history.py:UploadedFilesLookup, the interface of both lookups;parse_uploaded_files_record(), shared by both, so that they agree on what is uploaded (a file size larger than SQLite stores makes a record invalid). Records are written whole: to a temporary file, then moved in place, so that a reader never sees a partial record and an interrupted write doesn't lose the previous one (also true onmainbefore).upload.py:uploadandcheck_upload_historyuse the index and close it when done.uploadvalidates its options before opening the history.check_upload_historychanges nothing but the index (README and command reference). This changes what the docs promised; it's intended, as the Desktop Uploader runscheck_upload_historyon every file drop, where the index saves the most.Measurements
A synthetic history of 15,000 records × 100 files (1.5M files, 176 MB of records), looking up 200 new files:
The index file is 121 MB for 1.5M files.
Review
Before opening this PR, an independent review (another AI agent) checked parity with reading every record, multi-process stress (2 writers + 4 readers, final index equal to the records), interrupts (Ctrl-C, SIGKILL mid-build), rebuild and fallback. Its findings, and those of a later review (atomic record writes, change detection beyond modification time and size, an index left read-only), are fixed here: a lookup failing during a run now falls back instead of aborting the upload; out-of-range values make a record invalid instead of crashing; only corruption deletes the index; the index is closed when
uploadfails early; the write-ahead log size is capped. A second review of the fixes found no blockers. It also found that a record with an upload time out of range could makeuploadfail when skipping one of its files; that is fixed in #957.Not changed: a read-only upload history folder falls back to reading every record (as in #957), and WAL doesn't work across hosts on a network filesystem, which the upload history, kept on this computer, isn't meant for.
Known limits
Test plan
ruff check,ruff format --check,usort diff,mypy mapillary_tools tests/cli: clean.pytest tests: 1717 passed, 96 skipped (macOS, Python 3.12, ExifTool 13.60).tests/unit/test_history_index.py(21 tests, also each alone and after the CLI tests):uploadcloses the index when it fails; no history or nothing uploaded yet: nothing created.check_upload_historytests of Look files up in the upload history by size and fingerprint #957 run through the index.