Skip to content

Rerun only the failed tests on CI retries - #3178

Open
antonwolfy wants to merge 6 commits into
masterfrom
retry-last-failed-tests
Open

antonwolfy wants to merge 6 commits into
masterfrom
retry-last-failed-tests

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

The test steps wrapped in nick-fields/retry reran the whole suite on every attempt. A real failure cost a second full run (up to ~26 min for the all-integer-dtypes run) before the job went red, and since #3152 a tensor test killed by pytest-timeout reruns the whole tensor suite.

Each pytest command inside a retry step now passes --lf:

  • The first attempt has no recorded failures, so it runs everything (--lfnf defaults to all).
  • A retry runs only the tests that failed in the previous attempt.
  • An attempt killed by the step timeout, or a crashed pytest process, writes no cache, so the next attempt runs everything again.

Serial commands that a --lf retry follows also pass --continue-on-collection-errors. Without it a collection error stops the session before any test runs, and a passing retry of the broken module turns the job green without running the rest. The -n auto commands don't need it: xdist already runs the remaining tests after a collection error.

Covered: conda-package.yml (Linux and Windows, dpnp and tensor tests), conda-build-test.yml (conda-forge), check-onemath.yaml and the rerun steps of cron-run-tests.yaml. generate_coverage.yaml keeps full reruns, since a partial rerun would replace the coverage data of the full one.

A test that fails only after other tests have run passes when rerun on its own, so a retry can hide it. To keep such tests visible:

  • Before each --lf run, the cached failures are printed as a ::warning:: annotation naming the tests the retry reruns (up to 50). The first attempt has no cache and prints nothing. This works on fork PRs too.
  • In conda-package.yml the test jobs also upload that list as an artifact, and a new Report retried tests job posts it as one PR comment, grouped by job and suite and updated on each push. When a later push needs no retries, an existing comment is rewritten to say so. This job is the only one with pull-requests: write and, like the Array API comment, is skipped on fork PRs.

Checked locally with pytest 9.2.0, pytest-xdist 3.8.0 and pytest-timeout, on a package run through --pyargs as in CI, with one test that fails once and one that hangs once:

Attempt Result
1, no cache (-n 2 --lf --timeout=3 --timeout-method=thread) all 5 tests ran: 3 passed, 2 failed (one in the worker killed by the timeout)
2, same command only the 2 failed tests ran, both passed
3, same command no failures left in the cache, all 5 ran
after a run killed with SIGTERM no cache written, all 5 ran

The annotation and the report job's message were checked the same way, with a dpnp and a tensor test that fail once. The upload, the download and the comment itself only run in CI.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_12 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

coveralls commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.269%. remained the same — retry-last-failed-tests into master

@antonwolfy antonwolfy self-assigned this Nov 8, 2026
@antonwolfy
antonwolfy requested a balanced review from Copilot November 8, 2026 19:50
@antonwolfy antonwolfy added this to the 0.22.0 release milestone Nov 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Serial pytest attempts can stop on collection errors, allowing a successful --lf retry to pass CI without running the remaining suite; the affected full attempts need --continue-on-collection-errors.

1 open issue
What changed in this PR

This PR reduces CI retry time by using pytest’s last-failed cache instead of rerunning entire suites, building on the existing tensor-test timeout handling.

Changes:

  • Adds --lf to retry-wrapped pytest commands across package, conda-forge, oneMath, and scheduled workflows.
  • Documents the intended retry behavior while leaving coverage runs unchanged to preserve full-suite coverage data.
File Description
.github/​workflows/​cron-run-tests.yaml Limits Linux and Windows reruns to previously failed tests.
.github/​workflows/​conda-package.yml Adds last-failed selection to dpnp and tensor retries on Linux and Windows.
.github/​workflows/​conda-build-test.yml Adds last-failed selection to conda-forge test retries.
.github/​workflows/​check-onemath.yaml Adds last-failed selection to dpnp and tensor retries for both oneMath configurations.

Review effort: Balanced

Comment thread .github/workflows/conda-package.yml Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The targeted CI retry changes have no active review issues.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3178/index.html

@ndgrigorian

Copy link
Copy Markdown
Collaborator

@antonwolfy
One concern here is that some tests may only fail as part of a full suite run. It's probably fine either way (if this is consistent, we will notice) but it's something worth considering as unexpected by-product (creation of seemingly "flaky" tests)

@antonwolfy

Copy link
Copy Markdown
Contributor Author

@ndgrigorian, we probably need to post comment to the PR if there was the rerun triggered.
In any case the 1st run include the whole scope always.

The `nick-fields/retry` steps reran the whole suite on every attempt, so a real failure cost a second full run (up to ~26 min) before the job went red, and a test killed by `pytest-timeout` reran all tensor tests.

Each pytest command inside a retry step now passes `--lf`. The first attempt has no recorded failures and runs everything; a retry runs only the tests that failed. An attempt killed by the step timeout records nothing, so the next one runs all tests again. Coverage keeps full reruns, since a partial rerun would drop coverage data.
A collection error interrupts a serial pytest run before any test runs and records only the broken module, so a `--lf` retry ran just that module and could pass without the rest of the suite. With `--continue-on-collection-errors` the first attempt runs the other tests, and the retry covers their failures too. The `-n auto` runs don't need it, since xdist already runs the remaining tests after a collection error.
@antonwolfy
antonwolfy force-pushed the retry-last-failed-tests branch from 103bcde to 097a285 Compare November 16, 2026 12:22
Before each --lf pytest run, print the tests recorded in the last-failed cache as a ::warning:: annotation. A first attempt has no cache and prints nothing, so the warning appears only on retries, and a test that fails in the full run but passes on its own stays visible on the run summary, also for fork PRs.
The conda-package test jobs record the tests each --lf retry reruns and upload them as an artifact. A report job, the only one with pull-requests: write, collects them and posts or updates one PR comment listing them per job and suite, or rewrites an earlier comment when nothing was retried. Like the Array API comment it is skipped on fork PRs, where the warning annotations still name the tests.
@antonwolfy

antonwolfy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@ndgrigorian, a test that fails only after other tests have run would pass when rerun on its own, and the job would go green.

This matters most for the serial runs (the all-integer-dtypes jobs, oneMath, cron): their order is fixed, so a full rerun used to fail such a test again. With -n auto, xdist already spreads tests across workers differently on each run, so a full rerun would often have passed it too. A crash or a timeout kill writes no cache, so those retries still run the whole suite.

To keep such tests visible:

  • Before a retry, the tests it is about to rerun are printed as a warning annotation on the run, so they show up on the run summary. This also works on fork PRs.
  • On non-fork PRs, a Report retried tests job collects these lists from all conda-package test jobs and posts them as one PR comment, updated on each push, the same way the Array API results are posted.

If a test keeps showing up there, it is a candidate for an order-dependence issue.

The inline one-liner copied into four workflows and the report job's heredoc are replaced by .github/scripts/test_retries.py, with a record command run before each --lf attempt and a report command that composes the PR comment. Cron's sparse checkout and the report job now also check out .github/scripts.
They sat above the retry settings but described pytest flags far below them; the reasons are in the commit history and the PR description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants