Skip to content

Look files up in the upload history by size and fingerprint - #957

Merged
rtwojcik merged 3 commits into
mainfrom
feat/fingerprint-upload-history
Nov 19, 2026
Merged

rtwojcik merged 3 commits into
mainfrom
feat/fingerprint-upload-history

Conversation

@rtwojcik

@rtwojcik rtwojcik commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

check_upload_history and upload look files up in the upload history by an MD5 of the whole file, so every image, video and zip file is read in full first. On large drops from slow drives that is the main wait. The Desktop Uploader runs check_upload_history after processing to show duplicates, and a user with 167 dashcam clips (66 GB) on an SD card waited about an hour without the upload starting: one full read alone takes about 12–55 minutes there.

This PR replaces the MD5 lookup with one fast rule, the same in both commands:

  1. No uploaded file has the same size → the file is new. It is not read.
  2. Same size → read its fingerprint: a BLAKE2b hash of the file size and of 64 KiB from the start, the middle and the end of the file (the whole file below 192 KiB). Then look the fingerprint up.
  3. The fingerprint was uploaded → already uploaded. Otherwise → new.

Each upload records the size and fingerprint of its files. Because both commands apply the same rule, upload skips exactly the files that check_upload_history reports as uploaded.

Changes

  • fingerprint.py (new): fingerprint_file() and combine_fingerprints().
  • history.py:
    • New records in upload_history/fingerprints_v1/: one JSON file per uploaded sequence, video or zip file, with the name, size and fingerprint of each file, plus the run params and summary as before.
    • read_uploaded_files() reads them in one pass into the sizes and fingerprints uploaded.
    • A record of an upload that Mapillary didn't finish (cluster ID null in its summary) is not taken as uploaded: is_unfinished_upload from Don't take unfinished uploads in the upload history as uploaded #956 now applies to these records.
    • An upload time that can't be shown as a local time (out of range, NaN) is taken as not known.
    • The records by checksum and their functions (history_desc_path, read_history_record, write_history, find_uploaded_image_md5s) are removed.
  • check_upload_history:
    • Uses the rule for images, videos and zip files, and computes no checksum.
    • The JSON output keeps its fields. sequence_md5sum is now always empty; it is kept because the Desktop Uploader's parser requires it, and can be removed together with that parser later.
  • upload (also behind process_and_upload and video_process_and_upload):
    • The files to upload are found once, and the history records exactly those: the images uploaded, not every image of their sequence in the description file.
    • The duplicate check uses the rule.
    • Progress events keep a non-empty sequence_md5sum for images and videos: a key of the file paths (upload_key), the same in every run. The Desktop Uploader counts uploaded bytes by it; zip files still send their checksum.
    • upload_start is now emitted before the stream is read for the session key, so a duplicate is skipped without being read.
    • No full MD5 pass before uploading images and videos. The checksum cache that only served that pass is removed (create_checksum_cache, _update_md5sums).
    • Session keys (resume) are unchanged: still the MD5 of the bytes sent.
  • Docs: command reference updated.

Behavior changes

  • Image sequences: upload now skips a sequence when all its images were uploaded before, also as part of a larger or differently grouped sequence. Before, it skipped only the exact same sequence. This matches what check_upload_history already reported per image. A sequence with any new image is still uploaded whole.
  • Renamed files: videos and zip files are recognized after being renamed.
  • Files uploaded with earlier versions only have records by checksum, so they are not found and are uploaded once more. This is the price of not reading files in full; after that one upload they have a fingerprint record.
  • Folder vs zip: a sequence uploaded as a folder and later as a zip file, or the other way round, is no longer recognized across the two. Before, both shared the sequence checksum.
  • Older versions don't see the records written by this version.
  • Known limit: a file of the same size as an uploaded file that differs from it only outside the three sampled blocks is taken as uploaded. Captures from cameras don't differ that way; --reupload uploads such a file anyway. Documented in the command reference.
  • Scaling: upload and check_upload_history read every record by fingerprint at start. These records only start with this version; an index by file size for large histories can follow in a separate PR.

Why size + fingerprint is reliable

  • A file can only match a file of exactly the same size.
  • Every 64 KiB block of a compressed image or video differs from one capture to another.
  • An MP4 keeps its index, with the size and offset of every frame, at one end of the file: at the end in the GoPro sample files of the tests, at the start in "fast start" files. The start or end block covers it, and the middle block always falls on frame data.
  • A JPEG's EXIF is in the first block, so a metadata edit changes the fingerprint.
  • The same idea is used by the OpenSubtitles movie hash (size + first and last 64 KiB) and by imohash (start, middle and end + size).
  • BLAKE2b comes from the standard library, so there is no new dependency. Per 192 KiB sample on an Apple Silicon MacBook: MD5 0.30 ms, BLAKE2b 0.20 ms, BLAKE3 0.11 ms, XXH3 0.01 ms. Reading the sample takes far longer (about 5–15 ms from an SD card), so a faster hash would gain nothing.

Measurements

check_upload_history on main and on this branch. Same files, and an equivalent test upload history on each side: 200 sequences of 300 images and 200 videos, plus the uploads below (403 records by fingerprint, 414 by checksum). MacBook with an internal SSD, files in the page cache.

Files checked main This PR
22 new 150 MB videos (3.3 GB) 3.29 GB read, 5.9 s nothing read, 0.7 s
24 videos, 2 uploaded before 3.59 GB read, 5.9 s 2 fingerprints read, 0.8 s
200 images, 2 uploaded inside another sequence 0.60 GB read, 6.1 s 6 fingerprints read, 0.5 s
All of the above (4.2 GB) 4.19 GB read, 7.8 s 8 fingerprints read, 0.9 s
24 videos, all uploaded before 3.59 GB read, 6.5 s 24 fingerprints read (5 MB), 0.9 s
  • Every run found the same duplicates on both sides.
  • What remains on this branch is reading the upload history: 0.7 s for this one, up to 3 s the first time it is read from disk.
  • From an SD card at 20–90 MB/s, main would take about 45 s to 3.5 min for these 4.2 GB, and minutes to an hour for a large dashcam drop. This branch stays at seconds.

Relation to other PRs

Rebased on main after #952–#956.

Test plan

  • pytest tests: 1688 passed, 96 skipped (macOS, Python 3.12, ExifTool 13.60), on main b425836. With the last commit, on Linux: all pass except video sampling tests that fail the same way without these changes (that machine's ffmpeg).
  • ruff check mapillary_tools, ruff format --check mapillary_tools tests, usort diff mapillary_tools, mypy mapillary_tools tests/cli
  • New and updated tests:
    • fingerprint: only the three blocks and the size count; small files are hashed whole;
    • history records: round trip, latest upload time, invalid records skipped, upload times out of range taken as not known, no history;
    • unfinished uploads (Don't take unfinished uploads in the upload history as uploaded #956's tests, on records by fingerprint): such a record is not read, finished ones are (also without a cluster ID or from --dry_run), check_upload_history ignores it, and upload uploads the files again and replaces the record;
    • check_upload_history: files of new sizes not read, a file of the same size told apart, renamed files, a subset and a mixed part of an uploaded sequence, invalid records and file order, records by checksum not used, zip files, no history;
    • upload: images recorded and skipped the next time, in agreement with check_upload_history; a duplicate video skipped without the session key read, and uploaded again with --reupload; a zip file recorded and skipped the next time; only the images uploaded are recorded when the description file lists more; an image deleted after process is neither uploaded nor recorded, and the sequence is skipped the next time;
    • progress events: every event of an image sequence (including the ones of its images) and of a video carries the same non-empty sequence_md5sum. Replayed through a port of the Desktop Uploader's byte counter, a 10-image upload now shows the running total, as on main.

@meta-cla meta-cla Bot added the cla signed label Nov 17, 2026
@rtwojcik
rtwojcik marked this pull request as ready for review November 18, 2026 13:08
@rtwojcik
rtwojcik requested a review from caglarpir November 18, 2026 13:58
check_upload_history and upload no longer read whole files to compute MD5s.
A file is new if no uploaded file had its size; otherwise it is looked up by
a fingerprint: a BLAKE2b hash of its size and of 64 KiB from its start,
middle and end. Both commands apply the same rule, so upload skips exactly
the files that check_upload_history reports as uploaded. Uploads are
recorded by fingerprint; the records by checksum of earlier versions are no
longer read. Records of uploads that Mapillary didn't finish are not taken
as uploaded, as #956 does for the records by checksum.
@rtwojcik
rtwojcik force-pushed the feat/fingerprint-upload-history branch from 85372bb to 6714444 Compare November 18, 2026 14:31
…ents

Review fixes:
- The history step took every image of the description file with the
  sequence UUID, not the images uploaded: uploading some files of a
  folder recorded the whole sequence, and an image deleted after process
  made recording fail, so the sequence was uploaded on every run. upload
  now finds the files to upload once and records exactly those.
- Progress events of images and videos have a non-empty sequence_md5sum
  again, a key of the file paths, as the Desktop Uploader counts the bytes
  uploaded by it.
- The upload tests that check INFO lines set the log level.
- The command reference tells the fingerprint's known limit.
A record with an upload_end_time that can't be shown as a local time, like
1e30 or Infinity in a hand-edited record, made upload fail when it skipped
a file of that record. Such times are now taken as not known, like times
that are not numbers.
@rtwojcik
rtwojcik merged commit be537db into main Nov 19, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant