Skip to content

Don't take unfinished uploads in the upload history as uploaded - #956

Merged
caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:history-ignore-unfinished-uploads
Nov 18, 2026
Merged

caglarpir merged 1 commit into
mapillary:mainfrom
caglarpir:history-ignore-unfinished-uploads

Conversation

@caglarpir

Copy link
Copy Markdown
Contributor

Follow-up to #955. Until #955, when finish_upload answered without a cluster ID, as for an organization that the account is not a member of, upload wrote the files to the upload history with "cluster_id": null, although nothing was uploaded. #955 stops writing such records, but the ones already written still count as uploaded in three places:

  • the duplicate check of upload: UploadedAlready: Skipping ..., already uploaded,
  • check_upload_history,
  • the lookup of images uploaded in other sequences.

So files that hit this stay skipped after upgrading, unless uploaded with --reupload.

A record whose summary has "cluster_id": null is now not taken as uploaded in any of the three. upload uploads the files again, the new record replaces the old one, and it logs:

Not taken as uploaded: the upload in the history record .../upload_history/35/dd982be7bba1e07b367c88fdd650a8.json was not finished

Only an explicit null counts. Records written before 0.14.0 have no cluster ID at all and still count as uploaded, so old uploads are not uploaded again.

Checked on the history record written by the live test in #955: check_upload_history on main reports the image as already uploaded, and on this branch as not uploaded.

Tests: an unfinished record is not read, is skipped by the image lookup, gives already_uploaded: false in check_upload_history, and is uploaded again by upload (all fail on main). Records without a cluster ID, with one, or with the dry-run "0" are still read, and upload still skips a record without one.

Independent of #955: it merges cleanly with it and with #950 to #954.

When finish_upload answered without a cluster ID, as for an organization
that the account is not a member of, upload wrote the files to the upload
history with "cluster_id": null, although nothing was uploaded. mapillary#955 stops
writing such records, but the ones already written still made upload skip
the files as already uploaded, and check_upload_history report them so.

A record whose summary has the cluster ID null is now not taken as
uploaded, by the duplicate check of upload, by check_upload_history and
by the lookup of images uploaded in other sequences; upload logs that it
ignores it. Only an explicit null counts: records written before 0.14.0
have no cluster ID and still count as uploaded.
@meta-cla meta-cla Bot added the cla signed label Nov 17, 2026
rtwojcik added a commit that referenced this pull request Nov 18, 2026
When finish_upload answers without a cluster ID, nothing was uploaded:
upload doesn't record the files, and a record whose summary has the
cluster ID null is not read, as #956 does for the records by checksum.
@caglarpir
caglarpir merged commit b425836 into mapillary:main Nov 18, 2026
16 checks passed
rtwojcik added a commit that referenced this pull request Nov 18, 2026
check_upload_history and upload no longer read whole files to compute MD5s.
A file is new if no uploaded file had its size; otherwise it is looked up by
a fingerprint: a BLAKE2b hash of its size and of 64 KiB from its start,
middle and end. Both commands apply the same rule, so upload skips exactly
the files that check_upload_history reports as uploaded. Uploads are
recorded by fingerprint; the records by checksum of earlier versions are no
longer read. Records of uploads that Mapillary didn't finish are not taken
as uploaded, as #956 does for the records by checksum.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant