Repository navigation
Rerun only the failed tests on CI retries - #3178
antonwolfy wants to merge 6 commits into
Conversation
|
Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_12 ran successfully. |
There was a problem hiding this comment.
🟡 Changes recommended
Serial pytest attempts can stop on collection errors, allowing a successful --lf retry to pass CI without running the remaining suite; the affected full attempts need --continue-on-collection-errors.
1 open issue
What changed in this PR
This PR reduces CI retry time by using pytest’s last-failed cache instead of rerunning entire suites, building on the existing tensor-test timeout handling.
Changes:
- Adds
--lfto retry-wrapped pytest commands across package, conda-forge, oneMath, and scheduled workflows. - Documents the intended retry behavior while leaving coverage runs unchanged to preserve full-suite coverage data.
| File | Description |
|---|---|
.github/workflows/cron-run-tests.yaml |
Limits Linux and Windows reruns to previously failed tests. |
.github/workflows/conda-package.yml |
Adds last-failed selection to dpnp and tensor retries on Linux and Windows. |
.github/workflows/conda-build-test.yml |
Adds last-failed selection to conda-forge test retries. |
.github/workflows/check-onemath.yaml |
Adds last-failed selection to dpnp and tensor retries for both oneMath configurations. |
Review effort: Balanced
|
View rendered docs @ https://intelpython.github.io/dpnp/pull/3178/index.html |
|
@antonwolfy |
|
@ndgrigorian, we probably need to post comment to the PR if there was the rerun triggered. |
The `nick-fields/retry` steps reran the whole suite on every attempt, so a real failure cost a second full run (up to ~26 min) before the job went red, and a test killed by `pytest-timeout` reran all tensor tests. Each pytest command inside a retry step now passes `--lf`. The first attempt has no recorded failures and runs everything; a retry runs only the tests that failed. An attempt killed by the step timeout records nothing, so the next one runs all tests again. Coverage keeps full reruns, since a partial rerun would drop coverage data.
A collection error interrupts a serial pytest run before any test runs and records only the broken module, so a `--lf` retry ran just that module and could pass without the rest of the suite. With `--continue-on-collection-errors` the first attempt runs the other tests, and the retry covers their failures too. The `-n auto` runs don't need it, since xdist already runs the remaining tests after a collection error.
103bcde to
097a285
Compare
Before each --lf pytest run, print the tests recorded in the last-failed cache as a ::warning:: annotation. A first attempt has no cache and prints nothing, so the warning appears only on retries, and a test that fails in the full run but passes on its own stays visible on the run summary, also for fork PRs.
The conda-package test jobs record the tests each --lf retry reruns and upload them as an artifact. A report job, the only one with pull-requests: write, collects them and posts or updates one PR comment listing them per job and suite, or rewrites an earlier comment when nothing was retried. Like the Array API comment it is skipped on fork PRs, where the warning annotations still name the tests.
|
@ndgrigorian, a test that fails only after other tests have run would pass when rerun on its own, and the job would go green. This matters most for the serial runs (the all-integer-dtypes jobs, oneMath, cron): their order is fixed, so a full rerun used to fail such a test again. With To keep such tests visible:
If a test keeps showing up there, it is a candidate for an order-dependence issue. |
The inline one-liner copied into four workflows and the report job's heredoc are replaced by .github/scripts/test_retries.py, with a record command run before each --lf attempt and a report command that composes the PR comment. Cron's sparse checkout and the report job now also check out .github/scripts.
They sat above the retry settings but described pytest flags far below them; the reasons are in the commit history and the PR description.

The test steps wrapped in
nick-fields/retryreran the whole suite on every attempt. A real failure cost a second full run (up to ~26 min for the all-integer-dtypes run) before the job went red, and since #3152 a tensor test killed bypytest-timeoutreruns the whole tensor suite.Each pytest command inside a retry step now passes
--lf:--lfnfdefaults toall).Serial commands that a
--lfretry follows also pass--continue-on-collection-errors. Without it a collection error stops the session before any test runs, and a passing retry of the broken module turns the job green without running the rest. The-n autocommands don't need it: xdist already runs the remaining tests after a collection error.Covered:
conda-package.yml(Linux and Windows, dpnp and tensor tests),conda-build-test.yml(conda-forge),check-onemath.yamland the rerun steps ofcron-run-tests.yaml.generate_coverage.yamlkeeps full reruns, since a partial rerun would replace the coverage data of the full one.A test that fails only after other tests have run passes when rerun on its own, so a retry can hide it. To keep such tests visible:
--lfrun, the cached failures are printed as a::warning::annotation naming the tests the retry reruns (up to 50). The first attempt has no cache and prints nothing. This works on fork PRs too.conda-package.ymlthe test jobs also upload that list as an artifact, and a newReport retried testsjob posts it as one PR comment, grouped by job and suite and updated on each push. When a later push needs no retries, an existing comment is rewritten to say so. This job is the only one withpull-requests: writeand, like the Array API comment, is skipped on fork PRs.Checked locally with pytest 9.2.0, pytest-xdist 3.8.0 and pytest-timeout, on a package run through
--pyargsas in CI, with one test that fails once and one that hangs once:-n 2 --lf --timeout=3 --timeout-method=thread)SIGTERMThe annotation and the report job's message were checked the same way, with a dpnp and a tensor test that fail once. The upload, the download and the comment itself only run in CI.