Repository navigation
Don't take unfinished uploads in the upload history as uploaded - #956
Merged
caglarpir merged 1 commit intoNov 18, 2026
Merged
Conversation
When finish_upload answered without a cluster ID, as for an organization that the account is not a member of, upload wrote the files to the upload history with "cluster_id": null, although nothing was uploaded. mapillary#955 stops writing such records, but the ones already written still made upload skip the files as already uploaded, and check_upload_history report them so. A record whose summary has the cluster ID null is now not taken as uploaded, by the duplicate check of upload, by check_upload_history and by the lookup of images uploaded in other sequences; upload logs that it ignores it. Only an explicit null counts: records written before 0.14.0 have no cluster ID and still count as uploaded.
rtwojcik
added a commit
that referenced
this pull request
Nov 18, 2026
When finish_upload answers without a cluster ID, nothing was uploaded: upload doesn't record the files, and a record whose summary has the cluster ID null is not read, as #956 does for the records by checksum.
rtwojcik
added a commit
that referenced
this pull request
Nov 18, 2026
check_upload_history and upload no longer read whole files to compute MD5s. A file is new if no uploaded file had its size; otherwise it is looked up by a fingerprint: a BLAKE2b hash of its size and of 64 KiB from its start, middle and end. Both commands apply the same rule, so upload skips exactly the files that check_upload_history reports as uploaded. Uploads are recorded by fingerprint; the records by checksum of earlier versions are no longer read. Records of uploads that Mapillary didn't finish are not taken as uploaded, as #956 does for the records by checksum.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #955. Until #955, when
finish_uploadanswered without a cluster ID, as for an organization that the account is not a member of,uploadwrote the files to the upload history with"cluster_id": null, although nothing was uploaded. #955 stops writing such records, but the ones already written still count as uploaded in three places:upload:UploadedAlready: Skipping ..., already uploaded,check_upload_history,So files that hit this stay skipped after upgrading, unless uploaded with
--reupload.A record whose summary has
"cluster_id": nullis now not taken as uploaded in any of the three.uploaduploads the files again, the new record replaces the old one, and it logs:Only an explicit null counts. Records written before 0.14.0 have no cluster ID at all and still count as uploaded, so old uploads are not uploaded again.
Checked on the history record written by the live test in #955:
check_upload_historyon main reports the image as already uploaded, and on this branch as not uploaded.Tests: an unfinished record is not read, is skipped by the image lookup, gives
already_uploaded: falseincheck_upload_history, and is uploaded again byupload(all fail on main). Records without a cluster ID, with one, or with the dry-run"0"are still read, anduploadstill skips a record without one.Independent of #955: it merges cleanly with it and with #950 to #954.