Skip to content

feat: retry unprocessed requests in batch_add_requests - #1244

Open
cobanfurkanx wants to merge 1 commit into
apify:masterfrom
cobanfurkanx:feat/retry-unprocessed-batch-requests
Open

cobanfurkanx wants to merge 1 commit into
apify:masterfrom
cobanfurkanx:feat/retry-unprocessed-batch-requests

Conversation

@cobanfurkanx

Copy link
Copy Markdown
Contributor

Summary

  • Add DEFAULT_UNPROCESSED_RETRIES_BATCH_ADD_REQUESTS = 3 and DEFAULT_MIN_DELAY_BETWEEN_UNPROCESSED_REQUESTS_RETRIES = timedelta(milliseconds=500) in src/apify_client/_consts.py.
  • Update sync RequestQueueClient.batch_add_requests() and async RequestQueueClientAsync.batch_add_requests() (along with _batch_add_requests_worker, _serialize_requests, _compute_retry_delay_seconds, and _record_batch_attempt_result) in src/apify_client/_resource_clients/request_queue.py to retry unprocessed_requests with exponential backoff while preserving non-breaking default behavior and recording rate-limit rounds via self._http_client.stats.add_rate_limit_error(attempt + 1).
  • Update test_batch_processed_partially_{async,sync} and add test_batch_add_requests_retries_unprocessed_requests_{async,sync} in tests/unit/test_client_request_queue.py.

Closes #1134

Verification

  • pytest tests/unit/test_client_request_queue.py tests/unit/test_statistics.py: 63 passed ([impit] and [httpx2])
  • poe lint (ruff format --check && ruff check): 249 files formatted, all checks passed
  • ty check src tests/unit/test_client_request_queue.py & ty check --python-platform linux: all checks passed
  • poe check-docstrings: async and sync method docstrings are in sync

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

batch_add_requests() doesn't retry unprocessed requests

1 participant