You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(cuda.core): run cuMemPoolSetAccess in relaxed mode in PinnedMemoryResource - #3475
Since #3405, PinnedMemoryResource.__init__ without options calls cuMemPoolSetAccess to grant the current device read/write access to the driver's default host pool. The CUDA driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. This breaks code that constructs PinnedMemoryResource() while another thread holds an active global-mode graph capture — including PyTorch's torch.cuda.graph, which uses global mode by default.
The fix introduces MP_set_access in _memory_pool.pyx, wrapping cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode / _exit_relaxed_capture_mode helpers. This is the same pattern already used for pool attribute reads and writes and trim() since #2834. _PMR_init calls the new helper instead of the bare with nogil: HANDLE_RETURN(...) block.
The with-options path (cuMemPoolCreate) is intentionally unaffected; that call remains refused under global capture as documented in the 1.3.0 notes.
cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi — stub regenerated by pre-commit
cuda_core/tests/test_memory.py — add test_pmr_construction_during_capture_keeps_capture_valid (three capture combinations) and test_pmr_construction_restores_capture_mode, modelled on the existing test_mempool_calls_during_capture_keep_capture_valid and test_dmr_init_restores_capture_mode
cuda_core/docs/source/release/1.3.1-notes.rst — new release notes file with fix entry
Counterpart review (AGENTS.md dual-root requirement): This change is entirely in cuda_core/, which is single-sourced. The cuda_bindings_12 / cuda_bindings_13 roots are unaffected; cuMemPoolSetAccess wrappers there are unchanged and no behavioral change applies to them.
SYNCED blocks: No SYNCED blocks are touched by this diff.
…yResource
Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool. The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread. Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.
Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since NVIDIA#2834.
Call it from _PMR_init instead of the bare with-nogil block.
The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.
Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.
ClosesNVIDIA#3451
Creating a pinned memory resource without options no longer invalidates an ongoing CUDA graph capture, including captures running on another thread.
This fix applies to global and thread-local capture modes. The operation preserves the calling thread’s previous capture mode, including when construction occurs on a different thread from the active capture.
Default PinnedMemoryResource() construction now grants memory-pool access through a helper that temporarily uses relaxed capture mode and restores the calling thread’s prior mode. Tests cover capture validity and mode restoration during global and thread-local capture.
Changes: Pinned memory resource capture handling
Layer / File(s)
Summary
Relaxed-mode pool access helper cuda_core/cuda/core/_memory/_memory_pool.pxd, cuda_core/cuda/core/_memory/_memory_pool.pyx
A new helper calls cuMemPoolSetAccess in relaxed capture mode, restores the prior mode, and propagates errors from either operation.
Resource construction and capture tests cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi, cuda_core/tests/test_memory.py, cuda_core/docs/source/release/1.3.1-notes.rst
Default resource construction uses the helper. Documentation and release notes describe capture behavior. Tests check capture validity and mode restoration across capture modes and threads.
Assessment against linked issues
Objective
Addressed
Explanation
Constructing PinnedMemoryResource() without options does not affect a graph capture on another thread. [3451]
✅
Possibly related issues
NVIDIA/cuda-python#3415 — The PR changes how the access grant is made, but does not widen the grant to all host-pool-capable devices.
NVIDIA/cuda-python#3472 — The PR changes the default construction access call, but does not establish a cause for the reported out-of-memory failure or address the teardown workaround.
Suggested reviewers:saiyambharara
Change: Bug fix · Severity of issue fixed: Medium
Merge Risk: Low · up to 69c5a
The change stops default PinnedMemoryResource() construction from invalidating a graph capture on another thread. The remaining concern is small and confined to the new tests: when capture ends in an error, cleanup of the graph builder and resource is skipped. Fix it with try/finally blocks, but it should not block the merge.
Security architecture risk: Minimal · up to 69c5a
The change preserves the existing pool, device, and read/write permissions while preventing construction from invalidating concurrent graph capture. The inspected paths restore the previous capture mode before reporting driver errors. No introduced or worsened material security or ownership risk was established.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — The changed permission operation can affect the process-shared current host pool during capture, rather than only a private resource. Relative to the base, its timing expands but its target and permission scope do not: the same single descriptor grants read/write access to the current device under the same host-pool-support guard.
Trust Boundaries and Controls
inferred — The relaxed-mode exchange changes capture restrictions for the calling thread, not the selected pool, device identity, or access flags. The inspected change introduces no new caller-controlled permission descriptor or authorization authority.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
suggestion: Test non-default thread interaction modes.
begin_building(mode=mode) selects the capture mode. It does not set the calling thread's interaction mode. On a fresh thread, both cases therefore check restoration from GLOBAL. A helper that always restores GLOBAL would pass. (raw.githubusercontent.com)
Explicitly test prior interaction modes GLOBAL, THREAD_LOCAL, and RELAXED. Restore the original interaction mode in an outer finally. This will detect restoration to the wrong mode.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Reviewing files that changed from the base of the PR and between 0a0b675 and 69c5a4b.
Files selected for processing: 2
cuda_core/docs/source/release/1.3.1-notes.rst
cuda_core/tests/test_memory.py
Files skipped from review as they are similar to previous changes: 1
cuda_core/docs/source/release/1.3.1-notes.rst
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Stability & Availability · Quick win
important: Run resource cleanup even if end_building() raises.
An invalidated capture makes end_building() raise. Both cleanup blocks then skip gb.close() and mr.close(). This affects the failure these tests are intended to detect. The graph-builder contract explicitly permits this exception. (raw.githubusercontent.com)
Use nested try/finally blocks so both resources receive cleanup attempts without suppressing the test failure.
As per coding guidelines, “Release what you construct regardless.”
…yResource (NVIDIA#3475)
Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool. The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread. Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.
Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since NVIDIA#2834.
Call it from _PMR_init instead of the bare with-nogil block.
The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.
Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.
ClosesNVIDIA#3451
Co-authored-by: Andy Jost <ajost@nvidia.com>
(cherry picked from commit d94f19c)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bugSomething isn't workingcuda.coreEverything related to the cuda.core moduleP0High priority - Must do!to-be-backportedTrigger the bot to raise a backport PR upon merge
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3451
Since #3405,
PinnedMemoryResource.__init__without options callscuMemPoolSetAccessto grant the current device read/write access to the driver's default host pool. The CUDA driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. This breaks code that constructsPinnedMemoryResource()while another thread holds an active global-mode graph capture — including PyTorch'storch.cuda.graph, which uses global mode by default.The fix introduces
MP_set_accessin_memory_pool.pyx, wrappingcuMemPoolSetAccessin the existing_enter_relaxed_capture_mode/_exit_relaxed_capture_modehelpers. This is the same pattern already used for pool attribute reads and writes andtrim()since #2834._PMR_initcalls the new helper instead of the barewith nogil: HANDLE_RETURN(...)block.The with-options path (
cuMemPoolCreate) is intentionally unaffected; that call remains refused under global capture as documented in the 1.3.0 notes.Files changed:
cuda_core/cuda/core/_memory/_memory_pool.pxd— declareMP_set_accesscuda_core/cuda/core/_memory/_memory_pool.pyx— implementMP_set_accesswith relaxed-capture wrappercuda_core/cuda/core/_memory/_pinned_memory_resource.pyx— callMP_set_access; remove now-unusedas_cucimport; add one-sentence docstring notecuda_core/cuda/core/_memory/_pinned_memory_resource.pyi— stub regenerated by pre-commitcuda_core/tests/test_memory.py— addtest_pmr_construction_during_capture_keeps_capture_valid(three capture combinations) andtest_pmr_construction_restores_capture_mode, modelled on the existingtest_mempool_calls_during_capture_keep_capture_validandtest_dmr_init_restores_capture_modecuda_core/docs/source/release/1.3.1-notes.rst— new release notes file with fix entryCounterpart review (AGENTS.md dual-root requirement): This change is entirely in
cuda_core/, which is single-sourced. Thecuda_bindings_12/cuda_bindings_13roots are unaffected;cuMemPoolSetAccesswrappers there are unchanged and no behavioral change applies to them.SYNCED blocks: No SYNCED blocks are touched by this diff.
Checklist