Skip to content

Fix the OpenCL CPU compiler crash in the one-work-group radix sort - #3191

Open
ndgrigorian wants to merge 2 commits into
masterfrom
fix-radix-sort-cpu-crash
Open

ndgrigorian wants to merge 2 commits into
masterfrom
fix-radix-sort-cpu-crash

Conversation

@ndgrigorian

Copy link
Copy Markdown
Collaborator

This PR fixes a crash of the OpenCL CPU compiler in the one-work-group radix sort, which dpnp.sort, dpnp.argsort and their dpnp.tensor counterparts take for rows of 2049 to 16384 elements. Radix sort is the default for bool, int8, uint8, int16 and uint16, so a plain dpnp.sort of such rows can segfault on a CPU device, for example dpt.sort(dpt.zeros((1, 4096), dtype="i1", device="opencl:cpu")).

On a CPU device whose kernel gets sub-groups shorter than 16 (8 for the int8 kernels on an AVX2 CPU), the kernel puts a group_barrier after every unrolled counter update. That is the work-around for CMPLRLLVM-64592, added in IntelPython/dpctl@5125e11eb4. With 16 or more elements per work-item, those barriers crash the compiler in its sycl-kernel-reduce-cross-barrier-values pass (copyAndReplaceUses). Which data types crash depends on the compiler's heap state; once, int64 hung instead. The crash shows up with every OpenCL CPU runtime from 2024.18 (intel-opencl-rt 2025.0.4) to 2026.22 (2026.2.0).

The work-around is still needed: without it, the int8 radix argsort is unstable on the same runtimes. This PR replaces the barrier with a work-group atomic_fence. That keeps the int8 argsort stable, adds no barrier region, and so no longer crashes the compiler.

With a standalone SYCL reproducer, sort and stable argsort of bool, int8, uint8, int16, uint16, int32, int64, float32 and float64 match std::sort/std::stable_sort in both orders. That covers 1, 37 and 300 rows of every one-work-group size class, with full-range and low-cardinality data, on seven OpenCL CPU runtimes from 2025.0.4 to 2026.2.0. test_radix_sort_one_work_group covers rows of 257 to 16384 elements. Without the fix it segfaults on opencl:cpu for int8, uint8 and int32, and int64 times out; with it, it passes on opencl:cpu and level_zero:gpu, as do the sorting, top_k and unique tests.

GPUs don't take the work-around. Sort and argsort timings alternated between the two builds show no difference beyond noise on the iGPU (level_zero:gpu) or on opencl:cpu, at sizes the old build compiles.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

…ices rather than synchronize the work-group

On a CPU device whose kernel takes sub-groups shorter than 16, each counter
update was followed by a group_barrier, the work-around of CMPLRLLVM-64592
(int8 radix argsort comes out unstable). With 16 or more elements per
work-item, from rows of 2049 elements, these barriers crash the OpenCL CPU
compiler in its sycl-kernel-reduce-cross-barrier-values pass, so dpnp.sort and
dpnp.argsort of such rows segfault; bool, int8, uint8, int16 and uint16 take
the radix sort by default. Seen with every OpenCL CPU runtime from 2024.18 to
2026.22.

A work-group fence in place of the barrier still keeps int8 argsort stable and
adds no barrier region, so the compiler no longer crashes. Sort and argsort of
all integer and floating data types are correct on the same runtimes, and GPU
timings are unchanged since GPUs don't take the work-around.
// due to a bug in CPU device implementation, an additional
// synchronization is necessary for short sub-group sizes
// the CPU device compiler miscompiles the counting for short
// sub-groups unless each update is fenced (CMPLRLLVM-64592); a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the same as CMPLRLLVM-79305?

@github-actions

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3191/index.html

@github-actions

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev2=np2py314h8d9cdd5_8 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@coveralls

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.269%. remained the same — fix-radix-sort-cpu-crash into master

Comment on lines +420 to +421
@pytest.mark.parametrize("n", [257, 1025, 2049, 4096, 8193, 16384])
@pytest.mark.parametrize("dtype", ["?", "i1", "u1", "i2", "i4", "i8"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misses two size classes: 513..1024 and 4097..8192 (2049 and 4096 share one, 8193 and 16384 another). On master, int8 rows of 8192 crash the CPU compiler too. u2 takes the radix sort by default but is not here; on master a (2, 3000) uint16 sort spends over 10 minutes in the CPU compiler. Maybe [257, 513, 1025, 2049, 4097, 8193] and "u2"?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants