You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(samples): pick a greenContext SM split the driver accepts on pre-13.1 drivers - #3482
The cuda_core/greenContext sample could not select an SM split on a device whose SM count cannot be divided into two equal aligned halves, and failed with:
Error: could not find an SM split that the driver accepts on this device
(total SMs=14, min_partition_size=2).
Two causes, both in the sample:
It asked the driver for two unequal SM groups, SMResourceOptions(count=(long_count, critical_count)). On drivers older than 13.1, cuda.core splits SMs with cuDevSmResourceSplitByCount, which sizes every group from a single min_count and rejects unequal counts outright (CUDA 12 SM splitting only supports homogeneous count values). The probe caught that exception and reported it as a driver refusal, so auto selection only ever worked where an equal two-way split exists — on 14 SMs that would be 7+7, and 7 is not a multiple of min_partition_size 2.
A device that cannot host two SM partitions at all failed the sample instead of waiving it.
Changes
Reserve the critical kernel's partition with a single-group request and give the device remainder to the long-running kernel. This is the one request shape both driver split APIs accept (cuDevSmResourceSplit on driver 13.1+, cuDevSmResourceSplitByCount before it), and it is what the library's own tests use.
Probes now report the driver's last rejection instead of hiding it, and reject a split that does not produce exactly one partition.
--split A,B keeps its shape: B is reserved for the critical kernel and must be a multiple of min_partition_size, and A must be the SMs the critical partition leaves behind. Error messages name the accepted value instead of blaming the driver.
A device that accepts no partition exits with the negotiated sample-waiver code, matching the other cuda.core samples.
Also adds two GPU-free tests in cuda_core/tests/example_tests/test_sample_verification.py that run the sample's split selection against the pre-13.1 count-API rules (14-SM device, a device too small to split, and an allocation failure during probing), and registers greenContext in the waiver-exit-code list that the existing test enforces.
No public issue: this was reported internally on a Windows/sm_75 GTX 1650 machine (14 SMs, driver older than 13.1).
Sample end-to-end on a real GPU with both split paths: the pre-13.1 cuDevSmResourceSplitByCount path (forced by installing cuda.bindings 13.0.x) and the 13.1+ structured path → auto split: (28, 8), exit 0, measurable green-context speedup
ruff check and ruff format --check clean
The reported 14-SM sm_75 hardware was not available here, so the pre-13.1 branch was exercised through cuDevSmResourceSplitByCount on a different GPU rather than on the original machine. On the reporting machine, --split 12,2 / auto selection should now produce a 12+2 split; if the driver accepts no partition at all the sample now skips with the reason printed instead of failing.
Note: this touches cuda_core/tests/example_tests/test_sample_verification.py, which open PR #3455 also edits; expect a trivial import/test-tail conflict depending on merge order.
The greenContext sample asked the driver for two unequal SM groups with
SMResourceOptions(count=(long, critical)). On drivers older than 13.1,
cuda.core splits SMs with cuDevSmResourceSplitByCount, which sizes every
group from a single min_count and rejects the request with "CUDA 12 SM
splitting only supports homogeneous count values". The discovery probe
swallowed that exception and reported it as a driver refusal, so auto
selection only worked on devices that can express an equal two-way split.
On the 14-SM sm_75 part in NVBug 7106432 (7 SMs per side,
min_partition_size 2) no such pair exists, so the sample exited 1 and
test_samples.py failed.
Reserve the critical partition with a single-group request and give the
device remainder to the long kernel, which both driver split APIs accept.
Probes report the last driver reason instead of hiding it, --split keeps
its shape (CRITICAL is reserved and must be aligned, LONG is the
remainder), and a device that cannot host two SM partitions exits with the
sample waiver code instead of failing. That also brings the sample in line
with the negotiated EXIT_WAIVED convention the other samples follow.
Extend test_sample_verification.py with two GPU-free tests that drive the
sample's split selection against the count-API rules on a 14-SM device and
on one too small to split.
No actionable comments were generated in the recent review.
Summary
Summary by CodeRabbit
Bug Fixes
Automatic green-context split selection now uses the partition size reported by the driver and can try different alignment granularities to find an accepted split.
Explicit splits are checked against device partitioning requirements. If the driver provides a different remainder than requested, the sample reports the mismatch and may suggest an accepted split.
When no supported automatic split is found—or the driver version is unsupported—the sample exits with a waiver code instead of treating the condition as an ordinary failure.
Documentation
Clarified split requirements, automatic selection behavior, and waiver conditions.
The greenContext sample now probes one critical SM partition and uses the driver-reported remainder for the long partition. It validates explicit splits against that remainder and uses the configured sample-waiver exit code when automatic selection finds no accepted partition.
Changes: Green-context partitioning
Layer / File(s)
Summary
Automatic split discovery and waiver handling samples/cuda_core/greenContext/greenContext.py, samples/cuda_core/greenContext/README.md, cuda_core/tests/example_tests/test_sample_verification.py
Automatic selection probes candidate critical partitions and uses the driver-reported remainder. If no candidate is accepted, parsing exits with the sample-waiver code. The README and tests document and cover this behavior.
Explicit split validation and context creation samples/cuda_core/greenContext/greenContext.py, samples/cuda_core/greenContext/README.md, cuda_core/tests/example_tests/test_sample_verification.py
Explicit splits require an aligned critical count and a long count equal to the driver-reported remainder. Context creation requests the critical partition and uses the remainder. The README and tests describe and check these rules.
Possibly related issues
Issue 2765: The change reports the driver's rejection reason during split probing. It does not change _split_with_general_api, which the issue identifies as its root-cause path.
Suggested reviewers:danielfrg
Change: Bug fix
Merge Risk: Minimal · up to a8583
The greenContext sample now picks an SM split that pre-13.1 drivers accept and exits with the sample-waiver code when no partition can be hosted. No concrete merge-blocking risk remains. Only the PR's administrative metadata checks (assignee, labels, milestone) still need attention.
Security architecture risk: Minimal · up to a8583
The sample uses the existing driver-supported partitioning interface and checks the actual partition sizes before creating contexts. No introduced or worsened security or failure-containment risk was established.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
suggestion: Assert the negotiated waiver code instead of the initializer AST.
Adding "greenContext" to WAIVER_SAMPLE_NAMES applies the exact-expression check on EXIT_WAIVED to greenContext.py. An equivalent initializer, such as one using os.getenv, fails that check even when the exit code is correct. The cuda_core/tests/ guidance asks tests to pin observable behavior, not implementation.
Exclude greenContext from the exact-expression comparison. Cover its waiver code in the stand-in test instead. Call monkeypatch.setenv("CUDA_PYTHON_SAMPLE_WAIVER_EXIT_CODE", ...) with a non-default value before _load_green_context_sample() runs. Then assert that parse_split(None, device) exits with that code. The default code 2 cannot detect a hard-coded exit value, so the non-default value is required.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Catch only candidate rejections while probing SM partitions: a MemoryError
comes from cuda.core failing to allocate the split result, so it is a broken
environment rather than a refused partition, and must not end up as a sample
waiver.
Also pin the negotiated waiver code behaviorally: load the sample with a
non-default CUDA_PYTHON_SAMPLE_WAIVER_EXIT_CODE and assert that the
no-partition path exits with it, and add a test that an allocation failure
during probing propagates.
Addresses CodeRabbit review on PR NVIDIA#3482.
On the nitpick (WAIVER_SAMPLE_NAMES pins the EXIT_WAIVED initializer): I kept greenContext in the list, because that check is the repo-wide source convention every other waiving cuda_core sample follows — one module-level EXIT_WAIVED = int(os.environ.get("CUDA_PYTHON_SAMPLE_WAIVER_EXIT_CODE", "2")) and no literal sys.exit(2) — and this sample now matches it instead of inlining the expression in main().
I did take the substance of the suggestion in a858343: the no-partition test now sets CUDA_PYTHON_SAMPLE_WAIVER_EXIT_CODE=77 before loading the sample and asserts both EXIT_WAIVED == 77 and the exit code, so a hard-coded value can no longer pass. That covers negotiation behaviorally while the AST check keeps enforcing the shared idiom.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cuda.coreEverything related to the cuda.core module
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
cuda_core/greenContextsample could not select an SM split on a device whose SM count cannot be divided into two equal aligned halves, and failed with:Two causes, both in the sample:
SMResourceOptions(count=(long_count, critical_count)). On drivers older than 13.1,cuda.coresplits SMs withcuDevSmResourceSplitByCount, which sizes every group from a singlemin_countand rejects unequal counts outright (CUDA 12 SM splitting only supports homogeneous count values). The probe caught that exception and reported it as a driver refusal, so auto selection only ever worked where an equal two-way split exists — on 14 SMs that would be 7+7, and 7 is not a multiple ofmin_partition_size2.Changes
cuDevSmResourceSpliton driver 13.1+,cuDevSmResourceSplitByCountbefore it), and it is what the library's own tests use.--split A,Bkeeps its shape:Bis reserved for the critical kernel and must be a multiple ofmin_partition_size, andAmust be the SMs the critical partition leaves behind. Error messages name the accepted value instead of blaming the driver.cuda.coresamples.Also adds two GPU-free tests in
cuda_core/tests/example_tests/test_sample_verification.pythat run the sample's split selection against the pre-13.1 count-API rules (14-SM device, a device too small to split, and an allocation failure during probing), and registersgreenContextin the waiver-exit-code list that the existing test enforces.No public issue: this was reported internally on a Windows/sm_75 GTX 1650 machine (14 SMs, driver older than 13.1).
Testing
pytest cuda_core/tests/example_tests/test_sample_verification.py cuda_core/tests/test_green_context.py→ 76 passed, 3 skippedcuDevSmResourceSplitByCountpath (forced by installingcuda.bindings13.0.x) and the 13.1+ structured path →auto split: (28, 8), exit 0, measurable green-context speedupruff checkandruff format --checkcleanThe reported 14-SM sm_75 hardware was not available here, so the pre-13.1 branch was exercised through
cuDevSmResourceSplitByCounton a different GPU rather than on the original machine. On the reporting machine,--split 12,2/ auto selection should now produce a 12+2 split; if the driver accepts no partition at all the sample now skips with the reason printed instead of failing.Note: this touches
cuda_core/tests/example_tests/test_sample_verification.py, which open PR #3455 also edits; expect a trivial import/test-tail conflict depending on merge order.Checklist