Skip to content

ci: docs-only PRs run the full test matrix unless the title has [doc-only] or main has a green baseline #3480

Open

Description

@Andy-Jost

What happens

Docs-only PRs often run the full build and test matrix. Two mechanisms are supposed to prevent that, and both miss in common cases:

  1. The title tag. should-skip in .github/workflows/ci.yml skips the non-docs builds and all test jobs only when the PR title contains the literal, case-sensitive [doc-only]. Most docs PRs use a conventional prefix such as docs(cuda.core): ... instead, which does not match. The tag is not documented anywhere in the repo.
  2. The path planner. ci/tools/compute_ci_plan.py already treats <package>/docs/**, *.md and a few other paths as needing no tests. But it uses the changed paths only when the merge base has a successful main CI run with its artifacts. Without one, main() passes an empty path list and force_all is set ("this run will build and test everything").

Lately main rarely has that baseline: of the main push runs since 2026-10-17, 35 succeeded, 44 failed and 75 were cancelled. So an untagged docs PR now gets the full matrix.

The rule is the same for cuda.core, cuda.bindings and cuda.pathfinder (one ci.yml, one planner). The habit differs: cuda.bindings and cuda.pathfinder docs PRs mostly carry [doc-only], while cuda.core docs PRs mostly use docs(cuda.core): ....

Evidence (PRs opened since 2026-10-17)

Cost

A docs run that skips tests costs about 12-50 job-minutes (build-linux-64 for the docs, the docs build, the checks). A full run costs about 1,050 job-minutes, more than 20 times as much. It also occupies the shared GPU runners, so other PRs wait longer in the queue.

Options

  1. Detect docs-only changes from the paths, with or without a baseline. Always compute the changed paths against the merge base. If every path is one the planner already ignores, skip the tests even when there is no baseline. The [doc-only] tag becomes unnecessary. Drawback: the ignore list becomes the only safeguard, so a wrong entry would skip tests on real code. The same list already makes this decision whenever a baseline exists.
  2. Also accept the conventional docs(...): / doc(...): prefix, when the paths agree. Drawback: it keeps CI behavior tied to title conventions. It still misses untagged docs PRs with other titles (Bring over 13.4.4 release notes to main #3264, Omit autosummary pages in PR previews #3244, [doc-only] docs(release): tag cuda-core releases on a release branch #3431), and it needs the path check of option 1 anyway to be safe.
  3. A check that flags a mismatch between the title and the paths. For example, [doc-only] with code changes, or a docs-only change without the tag. Drawback: it adds one more check that authors must react to, and it does not save any runner time by itself.

I would do option 1, and honor [doc-only] only when the paths agree, so the tag can no longer skip tests on code changes. Option 1 fixes the problem at the planner, where the ignore list already lives, for all three packages and any title style. The [no-ci] escape hatch can stay as it is.

Activity

  1. changed the title [-]ci: docs-only PRs run the full test matrix unless the title has [doc-only] and main has a green baseline[/-] [+]ci: docs-only PRs run the full test matrix unless the title has [doc-only] or main has a green baseline[/+] on Nov 17, 2026
  2. added
    P2Low priority - Nice to have
    CI/CDCI/CD infrastructure
    and removed
    triageNeeds the team's attention
    on Nov 17, 2026
  3. self-assigned this
    on Nov 17, 2026
  4. added this to the CI milestone on Nov 17, 2026
  5. added theissue type on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP2Low priority - Nice to have

Type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions