Repository navigation
Look files up in the upload history by size and fingerprint - #957
Merged
Merged
Conversation
check_upload_history and upload no longer read whole files to compute MD5s. A file is new if no uploaded file had its size; otherwise it is looked up by a fingerprint: a BLAKE2b hash of its size and of 64 KiB from its start, middle and end. Both commands apply the same rule, so upload skips exactly the files that check_upload_history reports as uploaded. Uploads are recorded by fingerprint; the records by checksum of earlier versions are no longer read. Records of uploads that Mapillary didn't finish are not taken as uploaded, as #956 does for the records by checksum.
rtwojcik
force-pushed
the
feat/fingerprint-upload-history
branch
from
November 18, 2026 14:31
85372bb to
6714444
Compare
…ents Review fixes: - The history step took every image of the description file with the sequence UUID, not the images uploaded: uploading some files of a folder recorded the whole sequence, and an image deleted after process made recording fail, so the sequence was uploaded on every run. upload now finds the files to upload once and records exactly those. - Progress events of images and videos have a non-empty sequence_md5sum again, a key of the file paths, as the Desktop Uploader counts the bytes uploaded by it. - The upload tests that check INFO lines set the log level. - The command reference tells the fingerprint's known limit.
A record with an upload_end_time that can't be shown as a local time, like 1e30 or Infinity in a hand-edited record, made upload fail when it skipped a file of that record. Such times are now taken as not known, like times that are not numbers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check_upload_historyanduploadlook files up in the upload history by an MD5 of the whole file, so every image, video and zip file is read in full first. On large drops from slow drives that is the main wait. The Desktop Uploader runscheck_upload_historyafter processing to show duplicates, and a user with 167 dashcam clips (66 GB) on an SD card waited about an hour without the upload starting: one full read alone takes about 12–55 minutes there.This PR replaces the MD5 lookup with one fast rule, the same in both commands:
Each upload records the size and fingerprint of its files. Because both commands apply the same rule,
uploadskips exactly the files thatcheck_upload_historyreports as uploaded.Changes
fingerprint.py(new):fingerprint_file()andcombine_fingerprints().history.py:upload_history/fingerprints_v1/: one JSON file per uploaded sequence, video or zip file, with the name, size and fingerprint of each file, plus the run params and summary as before.read_uploaded_files()reads them in one pass into the sizes and fingerprints uploaded.nullin its summary) is not taken as uploaded:is_unfinished_uploadfrom Don't take unfinished uploads in the upload history as uploaded #956 now applies to these records.history_desc_path,read_history_record,write_history,find_uploaded_image_md5s) are removed.check_upload_history:sequence_md5sumis now always empty; it is kept because the Desktop Uploader's parser requires it, and can be removed together with that parser later.upload(also behindprocess_and_uploadandvideo_process_and_upload):sequence_md5sumfor images and videos: a key of the file paths (upload_key), the same in every run. The Desktop Uploader counts uploaded bytes by it; zip files still send their checksum.upload_startis now emitted before the stream is read for the session key, so a duplicate is skipped without being read.create_checksum_cache,_update_md5sums).Behavior changes
uploadnow skips a sequence when all its images were uploaded before, also as part of a larger or differently grouped sequence. Before, it skipped only the exact same sequence. This matches whatcheck_upload_historyalready reported per image. A sequence with any new image is still uploaded whole.--reuploaduploads such a file anyway. Documented in the command reference.uploadandcheck_upload_historyread every record by fingerprint at start. These records only start with this version; an index by file size for large histories can follow in a separate PR.Why size + fingerprint is reliable
Measurements
check_upload_historyonmainand on this branch. Same files, and an equivalent test upload history on each side: 200 sequences of 300 images and 200 videos, plus the uploads below (403 records by fingerprint, 414 by checksum). MacBook with an internal SSD, files in the page cache.mainmainwould take about 45 s to 3.5 min for these 4.2 GB, and minutes to an hour for a large dashcam drop. This branch stays at seconds.Relation to other PRs
Rebased on
mainafter #952–#956.check_upload_historytake MD5s from the upload checksum cache. With this PR neither command computes file MD5s, so the cache is removed. They conflict; this PR covers what Take check_upload_history checksums from the upload checksum cache #950 is for.finish_uploadfails without a cluster ID, so no record of such an upload is written, in either format.is_unfinished_upload) now applies to the records by fingerprint, and its tests are adapted to them; the records by checksum it covered are no longer read.Test plan
pytest tests: 1688 passed, 96 skipped (macOS, Python 3.12, ExifTool 13.60), onmainb425836. With the last commit, on Linux: all pass except video sampling tests that fail the same way without these changes (that machine's ffmpeg).ruff check mapillary_tools,ruff format --check mapillary_tools tests,usort diff mapillary_tools,mypy mapillary_tools tests/cli--dry_run),check_upload_historyignores it, anduploaduploads the files again and replaces the record;check_upload_history: files of new sizes not read, a file of the same size told apart, renamed files, a subset and a mixed part of an uploaded sequence, invalid records and file order, records by checksum not used, zip files, no history;upload: images recorded and skipped the next time, in agreement withcheck_upload_history; a duplicate video skipped without the session key read, and uploaded again with--reupload; a zip file recorded and skipped the next time; only the images uploaded are recorded when the description file lists more; an image deleted afterprocessis neither uploaded nor recorded, and the sequence is skipped the next time;sequence_md5sum. Replayed through a port of the Desktop Uploader's byte counter, a 10-image upload now shows the running total, as onmain.