Repository navigation
Conversation
…t pinning cuda-bindings (NVIDIA#3441) * docs(cuda.core): drop the Breaking Changes section name from the floor policy The release notes no longer have a Breaking Changes section for a floor move; 1.3.0 lists it under Highlights. Say only that the release notes list every move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cuda.core): advise against pinning cuda-bindings next to cuda-core Users who pin cuda-bindings themselves can conflict with the floor of a later cuda-core release. Tell them to depend on cuda-core with the cu12 or cu13 extra and let cuda-core choose the cuda-bindings version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 83a3422)
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com> (cherry picked from commit 1a22eec)
…NVIDIA#3436) * fix(cuda.core): serialize pinned-memory kernel-access test on Windows closes NVIDIA#3426 Add @thread_unsafe_on_windows to the pinned-memory kernel-access test. The test accesses host memory while parallel workers may execute GPU work; synchronizing its own stream does not coordinate those workers. This follows the guard used by similar tests on Windows and WSL. Windows CI is needed to confirm it resolves the reported crashes. The separate OOM failures remain under investigation. Validation: Ruff and git diff --check pass. * fix(cuda.core): report CUDA synchronization errors during test teardown Check the cuCtxSynchronize result in init_cuda so asynchronous errors surface during the triggering test's teardown instead of going unnoticed until a later test. Always attempt context cleanup via finally. Related to NVIDIA#3426. * fix(cuda.core): synchronize the bound context during test teardown Use device.sync() because the test wrapper may pop the current context before init_cuda teardown. This avoids spurious INVALID_CONTEXT errors while preserving synchronization error reporting and context cleanup. Add regression coverage for a missing current context and cleanup after synchronization failures. Related to NVIDIA#3426. * fix(cuda.core): serialize memory-pool watermark reset tests Mark test_mempool_watermark_reset thread_unsafe, matching test_mempool_trim. Both exercise the pool teardown and synchronization sequence associated with the suspected driver race in NVIDIA#3350. * fix(cuda.core): bypass access revocation when destroying host pools Skip peer-access revocation for locally created HOST and HOST_NUMA pools to investigate pinned-pool teardown OOMs and crashes. Continue destroying the pools and preserve cleanup for device, managed, and imported pools. This is a diagnostic workaround pending Windows CI validation. Related to NVIDIA#3426. * fix(cuda.core): preserve the first exception during fixture cleanup Keep setup or synchronization failures primary and attach subsequent cleanup failures as exception notes. Always attempt to pop the context. Add regression coverage for combined failures, including interrupts. * fix(cuda.core): limit host-pool cleanup workaround to pinned allocations HOST and HOST_NUMA also describe managed-pool preferences. Check the allocation type so the pinned-memory workaround preserves managed-pool peer-access revocation. Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * test(cuda.core): expose pinned allocation OOM and retain its traceback Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * ci: diagnose PR3436 pinned pool teardown on Windows Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * ci: move watermark investigation out of PR3436 Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * test(cuda.core): restore expected pinned allocation OOM handling Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * revert(cuda.core): defer pinned host pool cleanup workaround Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> --------- Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> (cherry picked from commit b715078)
…3471) * ci: exclude unavailable historical GitHub references Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * ci: require manual copy-pr-bot approval Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> --------- Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> (cherry picked from commit 3991a69)
Contributor
|
Andy-Jost
marked this pull request as draft
November 16, 2026 20:16
Contributor
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
…IA#3359) _smv_dtype_numpy_to_dlpack dispatched on numpy dtype kind, but ml_dtypes.bfloat16 has kind "V", so bf16 views fell through to the unsupported-dtype BufferError and could not be re-exported, even though the DLPack import path already accepts kDLBfloat and _tensor_map.pyx maps bfloat16. Detect bfloat16 up front and export it as kDLBfloat (16 bits). The new round-trip test goes through StridedMemoryView.from_dlpack so it cannot skip on stacks where NumPy's own DLPack support for extended dtypes is unavailable, unlike the existing parametrized round-trip test's NumPy-consumer probe. Signed-off-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com> Co-authored-by: Sainath-Reddy7 <180950590+Sainath-Reddy7@users.noreply.github.com> Co-authored-by: Sebastian Berg <sebastianb@nvidia.com> (cherry picked from commit d1d1457)
…me descriptor (NVIDIA#3450) * fix(cuda.core): make IPC imports wait for a pending release of the same descriptor deviceptr_import_ipc() shares one handle per descriptor because the driver's first free of an imported pointer unmaps every import of it (nvbug 5570902). The handle's deleter took ipc_import_mutex only after the last reference was gone. In that gap an import found the cache entry expired, re-imported the still-mapped pointer and registered a new entry. The deleter then erased that new entry and freed the pointer, which unmapped it under the new Buffer. The cache entry now outlives the last reference and only its own deleter erases it, under the mutex, together with the free. An import that finds an expired entry waits on a condition variable until that release is done, then imports fresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use CTAD for the ipc_import_mutex locks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2f1185e)
(cherry picked from commit 11cc666)
Signed-off-by: Rui Luo <ruluo@nvidia.com> (cherry picked from commit 7076a67)
GPUDirect RDMA is not supported on WSL, but some drivers report the capability there, and the RDMA-enabled VMM allocation in test_vmm_allocator_policy_configuration then fails. Skip the test on WSL; the capability checks stay as they are. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 36bc530)
…ce fix (NVIDIA#3479) * docs(cuda.core): add 1.3.0 release note for the IPC import/release race fix Follow-up to NVIDIA#3450, which merged without a release note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(cuda.core): scope the IPC race release note to memory pool buffers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit de4c60d)
) * Fix conditional graph lifetime tests to use supported nodes Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> * ci: require manual copy-pr-bot approval Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> --------- Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com> Co-authored-by: Andy Jost <ajost@nvidia.com> (cherry picked from commit b80866b)
…yResource (NVIDIA#3475) Since NVIDIA#3405, PinnedMemoryResource.__init__ without options calls cuMemPoolSetAccess to grant the current device read/write access to the driver's default pool. The driver refuses that call, and invalidates the capture, while a global-mode capture is active on any thread. Users of PyTorch's torch.cuda.graph (which defaults to global mode) therefore see CUDA_ERROR_STREAM_CAPTURE_UNSUPPORTED on construction and CUDA_ERROR_STREAM_CAPTURE_INVALIDATED on end_building. Fix: introduce MP_set_access in _memory_pool.pyx, wrapping cuMemPoolSetAccess in the existing _enter_relaxed_capture_mode / _exit_relaxed_capture_mode helpers, following the identical pattern used by pool attribute reads and writes and trim() since NVIDIA#2834. Call it from _PMR_init instead of the bare with-nogil block. The with-options path (cuMemPoolCreate) is unaffected; that call is still refused under global capture as documented in the 1.3.0 notes. Add tests for the three capture combinations (global+same_thread, global+other_thread, thread_local+same_thread) and a capture-mode restore check modelled on test_dmr_init_restores_capture_mode. Closes NVIDIA#3451 Co-authored-by: Andy Jost <ajost@nvidia.com> (cherry picked from commit d94f19c)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports the cuda.core 1.3.1 changes from
maintorelease/cuda-core-v1.3.x, in the order they merged onmain:cuda-bindings(docs only; cuda.core: carry the cuda-bindings floors through minimal-sysctk12/13 extras #3460'sinstall.rstchange builds on it).thread_unsafe(test only).1.3.1-notes.rst).cuMemPoolSetAccessin relaxed capture mode inPinnedMemoryResource, with its tests and release note (fixes [BUG]: Constructing PinnedMemoryResource() invalidates a global-mode graph capture on another thread #3451).Each commit is a
git cherry-pick -x. Please keep the(cherry picked from commit ...)lines in the squash message.More 1.3.1 changes will be added here as they merge on
main. Part of #3459.🤖 Generated with Claude Code