Skip to content

Build and test conda packages and wheels - #294

Open
jharlow-intel wants to merge 1 commit into
mainfrom
ci/intel-python-ci
Open

jharlow-intel wants to merge 1 commit into
mainfrom
ci/intel-python-ci

Conversation

@jharlow-intel

@jharlow-intel jharlow-intel commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Adds ci/ and .github/workflows/intel-ci.yml, which build the conda packages and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment. The recipe skips macOS, as conda-forge/mkl_umath-feedstock does.

  • conda-recipe/ is now the only recipe: the conda-forge one that was conda-recipe-cf/. conda-package.yml and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the recipe.
  • The conda packages build Python 3.10 to 3.15 plus 3.14t, from the python: list in conda_build_config.yaml. The wheels follow the pyproject.toml classifiers: 3.10 to 3.15 plus 3.14t and 3.15t.
  • Wheels build with cibuildwheel and icx: on Linux from Intel's oneAPI yum repository, on Windows from conda-forge's dpcpp_win-64, which ci/install_icx.py installs. They leave MKL and the compiler runtime out. ci/build.py adds their requirements to each wheel: mkl from the mkl-devel version they were built with up to its next major, through [tool.gh-ci] pin-compatible, and intel-openmp and intel-cmplr-lib-rt >=2026.2,<2027, through wheel-requires. icx links the Linux libraries with full RELRO, which ci/build.py checks, along with a library search path inside $ORIGIN.
  • meson.build adds Library to CMake's search path on Windows, where mkl-devel puts its CMake config.
  • ci/test.py runs pytest --pyargs mkl_umath on the installed package.
  • Each test job runs for every package that built, even when another build failed. A pull request builds the oldest and newest Python plus the newest free-threaded one; main and tags build them all.
  • conda-build records the source commit in each package's info/about.json.
  • ci/benchmark.py runs the asv suite in benchmarks/ against the installed package: one quick sample of each benchmark on a pull request, a full measurement on a tag.

Miniforge is pinned to 26.7.2-0, as 26.9.4-0 makes setup-miniconda's conda init fail on Windows runners.

Validation: run 42669879582 on d7e1dc3 builds and tests the reduced pull request matrix on Linux and Windows. Not run yet: the full matrix, which only main and tags build.

@jharlow-intel
jharlow-intel marked this pull request as ready for review October 23, 2026 20:42
@jharlow-intel jharlow-intel self-assigned this Oct 28, 2026
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 12 times, most recently from 02315bd to f53b1ce Compare November 3, 2026 18:00
@jharlow-intel jharlow-intel changed the title Build and test conda-recipe-cf for Intel Python CI Build and test conda packages and wheels Nov 5, 2026
@jharlow-intel
jharlow-intel force-pushed the ci/intel-python-ci branch 4 times, most recently from d7e1dc3 to 96e092f Compare November 9, 2026 21:50
Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds
conda-recipe/ with conda-build and the wheels with cibuildwheel, for every
platform and Python version the recipe and pyproject.toml declare, and tests
each package in a clean environment with ci/test.py.

conda-recipe/ is now the only recipe, the conda-forge one that was
conda-recipe-cf/; the Intel-channel recipe, conda-package.yml, which copied
wheels out through WHEELS_OUTPUT_FOLDER, and conda-package-cf.yml are removed,
as intel-ci.yml builds and tests the same recipe, running the test jobs for
each package that built even when another build failed.
conda_build_config.yaml now lists the Python versions.

The wheels link MKL's layers, the Intel OpenMP runtime and icx's runtime.
[build-system] requires mkl-devel, and [tool.gh-ci] pin-compatible maps mkl
to it: ci/build.py adds Requires-Dist mkl >= the mkl-devel version python -m
build installed, < its next major, the way conda's run_exports pin it, and
fails a build that logs no version. intel-openmp and intel-cmplr-lib-rt come
from [tool.gh-ci] wheel-requires: in [project] they would fail the recipe's
pip check, as conda-forge's packages have no pip metadata. ci/build.py takes
METADATA's line ending from the headers. The repair commands leave MKL's and
the compiler runtime's libraries out, and the Linux wheels are tagged
manylinux_2_28 only, like the mkl wheel.

icx already links the Linux libraries with full RELRO and a stack protector,
so the wheels need no extra flags. ci/build.py refuses a wheel whose ELF
files lack full RELRO. It keeps an RPATH or RUNPATH entry only when it is
$ORIGIN or a directory under it, as meson-python keeps the isolated build
env's absolute lib directory, and refuses a file left with none. It rewrites
the wheels in scratch, so a refused wheel never reaches --output, and
refuses two built packages with the same file name, which would overwrite
each other there.

conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions
sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda
package records the commit it was built from in info/about.json.

ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip
users install it. Every python: entry in conda_build_config.yaml names its
ABI, which ci/matrix.py requires. ci/matrix.py reads x86 as any x86 CPU and
osx64 as undefined, as conda-build does, and refuses a conda-build skip that
uses osx64. A pull request also builds and tests the newest free-threaded
version, and every push and tag run gets a concurrency group of its own.
Comment thread pyproject.toml
# What the wheels need at run time, which conda packages get from the recipe:
# mkl >= the mkl-devel version the build used, < its next major
pin-compatible = { mkl = "mkl-devel" }
wheel-requires = ["intel-openmp >=2026.2,<2027", "intel-cmplr-lib-rt >=2026.2,<2027"]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux icx isn't pinned (dnf install intel-oneapi-compiler-dpcpp-cpp, compiler/latest), but on Windows it's 2026.2.0, and the runtime requirements here are hardcoded to >=2026.2,<2027. With the next oneAPI release the Linux wheels will build with a newer icx and still allow the old runtime. dpnp#3151 pins icx to 2026.2 and gets these from the compiler version through pin-compatible. Could we do the same here?

Comment thread conda-recipe/meta.yaml
@@ -1,18 +1,22 @@
{% set version = "0.6.0" %}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other repos take the version from GIT_DESCRIBE_TAG/GIT_DESCRIBE_NUMBER, and so did the old recipe. With 0.6.0 and build 0 hardcoded, every build from main looks the same as the 0.6.0 release. The "takes its version from git describe" comment in intel-ci.yml is also wrong now.

Comment thread conda-recipe/build.sh
--no-index \
--prefix "${PREFIX}" \
CC=icx CXX=icpx $PYTHON -m pip install --no-build-isolation --no-deps \
-Csetup-args="-Dmkl_threading=gnu_thread" \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Linux conda package now uses gnu_thread with llvm-openmp, while the old recipe, Windows and the wheels use intel_thread with libiomp5. Is that intended? If these packages go to the Intel channel, it might be worth mentioning in the description.

Comment thread pyproject.toml
[tool.cython-lint]
ignore = ['E722'] # do not use bare 'except'
max-line-length = 80
[tool.cibuildwheel]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[tool.cython-lint] got removed here. Was that on purpose? cython-lint still runs in pre-commit, now with its default settings.

conda-remove-defaults: "true"

- name: Install conda-build
run: conda install -y -q -n base --override-channels -c conda-forge conda-build

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mkl-service pins conda-build=26.9.1. Should we pin it here too?

Comment thread conda-recipe/meta.yaml
- {{ pin_compatible('mkl') }}
- {{ pin_compatible('intel-openmp') }}
- {{ pin_compatible('intel-cmplr-lib-rt') }}
# NumPy 1 vs 2 loops are chosen at compile time (USE_NUMPY_2), so a build

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

USE_NUMPY_2 was removed in gh-289, so this comment is out of date. Same for "CI overrides it with --numpy" in conda_build_config.yaml, since nothing passes --numpy now.

Comment thread ci/config.yml
wheel: "mkl_umath {platform} Wheels Python {python}"
test:
requires: [pytest]
benchmark:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intel-ci.yml doesn't run benchmark.py or read this section. Could ci/README.md mention that, like in dpnp#3151? The description also makes it sound like PRs run the benchmarks.

Comment thread AGENTS.md
- **Workflows:** `.github/workflows/`
- `conda-package.yml` — main conda build/test pipeline
- `conda-package-cf.yml` — conda-forge-oriented build/test pipeline
- `intel-ci.yml` — Intel CI: builds and tests the conda packages and the wheels; see `ci/README.md`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing tests against the Intel channel anymore, but line 80 still says Intel's NumPy is required. Maybe update it and mention in the description that the Intel channel build is gone?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants