Skip to content

fix(cuda.core): run cuMemPoolSetAccess in relaxed mode in PinnedMemoryResource - #3475

Merged
Andy-Jost merged 3 commits into
NVIDIA:mainfrom
juenglin:fix/pmr-global-capture
Nov 17, 2026
Merged

Andy-Jost merged 3 commits into
NVIDIA:mainfrom
juenglin:fix/pmr-global-capture

Conversation

@juenglin

Copy link
Copy Markdown
Contributor

closes #3451

Since #3405, PinnedMemoryResource.__init__ without options calls cuMemPoolSetAccess to grant the current device read/write access to the driver's default host pool. The CUDA driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. This breaks code that constructs PinnedMemoryResource() while another thread holds an active global-mode graph capture — including PyTorch's torch.cuda.graph, which uses global mode by default.

The fix introduces MP_set_access in _memory_pool.pyx, wrapping cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode / _exit_relaxed_capture_mode helpers. This is the same pattern already used for pool attribute reads and writes and trim() since #2834. _PMR_init calls the new helper instead of the bare with nogil: HANDLE_RETURN(...) block.

The with-options path (cuMemPoolCreate) is intentionally unaffected; that call remains refused under global capture as documented in the 1.3.0 notes.

Files changed:

  • cuda_core/cuda/core/_memory/_memory_pool.pxd — declare MP_set_access
  • cuda_core/cuda/core/_memory/_memory_pool.pyx — implement MP_set_access with relaxed-capture wrapper
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx — call MP_set_access; remove now-unused as_cu cimport; add one-sentence docstring note
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi — stub regenerated by pre-commit
  • cuda_core/tests/test_memory.py — add test_pmr_construction_during_capture_keeps_capture_valid (three capture combinations) and test_pmr_construction_restores_capture_mode, modelled on the existing test_mempool_calls_during_capture_keep_capture_valid and test_dmr_init_restores_capture_mode
  • cuda_core/docs/source/release/1.3.1-notes.rst — new release notes file with fix entry

Counterpart review (AGENTS.md dual-root requirement): This change is entirely in cuda_core/, which is single-sourced. The cuda_bindings_12 / cuda_bindings_13 roots are unaffected; cuMemPoolSetAccess wrappers there are unchanged and no behavioral change applies to them.

SYNCED blocks: No SYNCED blocks are touched by this diff.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…yResource

Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool.  The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread.  Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.

Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since NVIDIA#2834.
Call it from _PMR_init instead of the bare with-nogil block.

The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.

Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.

Closes NVIDIA#3451
@juenglin juenglin added this to the cuda.core 1.3.1 milestone Nov 16, 2026
@juenglin juenglin added bug Something isn't working P0 High priority - Must do! cuda.core Everything related to the cuda.core module to-be-backported Trigger the bot to raise a backport PR upon merge labels Nov 16, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Nov 16, 2026
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 1542aa0

@juenglin
juenglin requested a review from Andy-Jost November 16, 2026 21:20
@juenglin
juenglin marked this pull request as ready for review November 16, 2026 21:21
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • Bug Fixes
    • Creating a pinned memory resource without options no longer invalidates an ongoing CUDA graph capture, including captures running on another thread.
    • This fix applies to global and thread-local capture modes. The operation preserves the calling thread’s previous capture mode, including when construction occurs on a different thread from the active capture.
    • Default begin_building() captures remain unaffected.

Default PinnedMemoryResource() construction now grants memory-pool access through a helper that temporarily uses relaxed capture mode and restores the calling thread’s prior mode. Tests cover capture validity and mode restoration during global and thread-local capture.

Changes: Pinned memory resource capture handling

Layer / File(s) Summary
Relaxed-mode pool access helper
cuda_core/cuda/core/_memory/_memory_pool.pxd, cuda_core/cuda/core/_memory/_memory_pool.pyx
A new helper calls cuMemPoolSetAccess in relaxed capture mode, restores the prior mode, and propagates errors from either operation.
Resource construction and capture tests
cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx, cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi, cuda_core/tests/test_memory.py, cuda_core/docs/source/release/1.3.1-notes.rst
Default resource construction uses the helper. Documentation and release notes describe capture behavior. Tests check capture validity and mode restoration across capture modes and threads.

Assessment against linked issues

Objective Addressed Explanation
Constructing PinnedMemoryResource() without options does not affect a graph capture on another thread. [3451] ✅

Possibly related issues

  • NVIDIA/cuda-python#3415 — The PR changes how the access grant is made, but does not widen the grant to all host-pool-capable devices.
  • NVIDIA/cuda-python#3472 — The PR changes the default construction access call, but does not establish a cause for the reported out-of-memory failure or address the teardown workaround.

Suggested reviewers: saiyambharara

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: Low · up to 69c5a

The change stops default PinnedMemoryResource() construction from invalidating a graph capture on another thread. The remaining concern is small and confined to the new tests: when capture ends in an error, cleanup of the graph builder and resource is skipped. Fix it with try/finally blocks, but it should not block the merge.

Security architecture risk: Minimal · up to 69c5a

The change preserves the existing pool, device, and read/write permissions while preventing construction from invalidating concurrent graph capture. The inspected paths restore the previous capture mode before reporting driver errors. No introduced or worsened material security or ownership risk was established.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed permission operation can affect the process-shared current host pool during capture, rather than only a private resource. Relative to the base, its timing expands but its target and permission scope do not: the same single descriptor grants read/write access to the current device under the same host-pool-support guard.

Trust Boundaries and Controls

  • inferred — The relaxed-mode exchange changes capture restrictions for the calling thread, not the selected pool, device identity, or access flags. The inspected change introduces no new caller-controlled permission descriptor or authorization authority.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick comments: 1

cuda_core/tests/test_memory.py: 1

3135-3137: Trivial · Maintainability & Code Quality · Quick win

suggestion: Test non-default thread interaction modes.

begin_building(mode=mode) selects the capture mode. It does not set the calling thread's interaction mode. On a fresh thread, both cases therefore check restoration from GLOBAL. A helper that always restores GLOBAL would pass. (raw.githubusercontent.com)

Explicitly test prior interaction modes GLOBAL, THREAD_LOCAL, and RELAXED. Restore the original interaction mode in an outer finally. This will detect restoration to the wrong mode.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 3991a69 and 1542aa0.

Files selected for processing: 6

  • cuda_core/cuda/core/_memory/_memory_pool.pxd
  • cuda_core/cuda/core/_memory/_memory_pool.pyx
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyi
  • cuda_core/cuda/core/_memory/_pinned_memory_resource.pyx
  • cuda_core/docs/source/release/1.3.1-notes.rst
  • cuda_core/tests/test_memory.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@github-actions

Copy link
Copy Markdown
Contributor

@Andy-Jost Andy-Jost left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Andy-Jost
Andy-Jost enabled auto-merge (squash) November 16, 2026 23:46
@Andy-Jost

Copy link
Copy Markdown
Contributor

/ok to test 0a0b675

# Conflicts:
#	cuda_core/docs/source/release/1.3.1-notes.rst
@Andy-Jost

Copy link
Copy Markdown
Contributor

/ok to test 69c5a4b

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0a0b675 and 69c5a4b.

Files selected for processing: 2

  • cuda_core/docs/source/release/1.3.1-notes.rst
  • cuda_core/tests/test_memory.py
Files skipped from review as they are similar to previous changes: 1

  • cuda_core/docs/source/release/1.3.1-notes.rst

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

pool.submit(construct).result()
finally:
# Raises if construction invalidated the capture.
gb.end_building()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Stability & Availability · Quick win

important: Run resource cleanup even if end_building() raises.

An invalidated capture makes end_building() raise. Both cleanup blocks then skip gb.close() and mr.close(). This affects the failure these tests are intended to detect. The graph-builder contract explicitly permits this exception. (raw.githubusercontent.com)

Use nested try/finally blocks so both resources receive cleanup attempts without suppressing the test failure.

As per coding guidelines, “Release what you construct regardless.”

Also applies to: 3143-3143

Source: Coding guidelines

@Andy-Jost
Andy-Jost merged commit d94f19c into NVIDIA:main Nov 17, 2026
121 checks passed
Andy-Jost added a commit to Andy-Jost/cuda-python that referenced this pull request Nov 17, 2026
…yResource (NVIDIA#3475)

Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls
cuMemPoolSetAccess to grant the current device read/write access to
the driver's default pool.  The driver refuses that call, and
invalidates the capture, while a global-mode capture is active on any
thread.  Users of PyTorch's torch.cuda.graph (which defaults to global
mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on
construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building.

Fix: introduce MP_set_access in _memory_pool.pyx, wrapping
cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode /
_exit_relaxed_capture_mode helpers, following the identical pattern
used by pool attribute reads and writes and trim() since NVIDIA#2834.
Call it from _PMR_init instead of the bare with-nogil block.

The with-options path (cuMemPoolCreate) is unaffected; that call is
still refused under global capture as documented in the 1.3.0 notes.

Add tests for the three capture combinations (global+same_thread,
global+other_thread, thread_local+same_thread) and a capture-mode
restore check modelled on test_dmr_init_restores_capture_mode.

Closes NVIDIA#3451

Co-authored-by: Andy Jost <ajost@nvidia.com>
(cherry picked from commit d94f19c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.core Everything related to the cuda.core module P0 High priority - Must do! to-be-backported Trigger the bot to raise a backport PR upon merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Constructing PinnedMemoryResource() invalidates a global-mode graph capture on another thread

2 participants