Repository navigation
Build and test conda packages and wheels - #294
jharlow-intel wants to merge 1 commit into
Conversation
02315bd to
f53b1ce
Compare
d7e1dc3 to
96e092f
Compare
Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds conda-recipe/ with conda-build and the wheels with cibuildwheel, for every platform and Python version the recipe and pyproject.toml declare, and tests each package in a clean environment with ci/test.py. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/; the Intel-channel recipe, conda-package.yml, which copied wheels out through WHEELS_OUTPUT_FOLDER, and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the same recipe, running the test jobs for each package that built even when another build failed. conda_build_config.yaml now lists the Python versions. The wheels link MKL's layers, the Intel OpenMP runtime and icx's runtime. [build-system] requires mkl-devel, and [tool.gh-ci] pin-compatible maps mkl to it: ci/build.py adds Requires-Dist mkl >= the mkl-devel version python -m build installed, < its next major, the way conda's run_exports pin it, and fails a build that logs no version. intel-openmp and intel-cmplr-lib-rt come from [tool.gh-ci] wheel-requires: in [project] they would fail the recipe's pip check, as conda-forge's packages have no pip metadata. ci/build.py takes METADATA's line ending from the headers. The repair commands leave MKL's and the compiler runtime's libraries out, and the Linux wheels are tagged manylinux_2_28 only, like the mkl wheel. icx already links the Linux libraries with full RELRO and a stack protector, so the wheels need no extra flags. ci/build.py refuses a wheel whose ELF files lack full RELRO. It keeps an RPATH or RUNPATH entry only when it is $ORIGIN or a directory under it, as meson-python keeps the isolated build env's absolute lib directory, and refuses a file left with none. It rewrites the wheels in scratch, so a refused wheel never reaches --output, and refuses two built packages with the same file name, which would overwrite each other there. conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda package records the commit it was built from in info/about.json. ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip users install it. Every python: entry in conda_build_config.yaml names its ABI, which ci/matrix.py requires. ci/matrix.py reads x86 as any x86 CPU and osx64 as undefined, as conda-build does, and refuses a conda-build skip that uses osx64. A pull request also builds and tests the newest free-threaded version, and every push and tag run gets a concurrency group of its own.
96e092f to
95766aa
Compare
| # What the wheels need at run time, which conda packages get from the recipe: | ||
| # mkl >= the mkl-devel version the build used, < its next major | ||
| pin-compatible = { mkl = "mkl-devel" } | ||
| wheel-requires = ["intel-openmp >=2026.2,<2027", "intel-cmplr-lib-rt >=2026.2,<2027"] |
There was a problem hiding this comment.
On Linux icx isn't pinned (dnf install intel-oneapi-compiler-dpcpp-cpp, compiler/latest), but on Windows it's 2026.2.0, and the runtime requirements here are hardcoded to >=2026.2,<2027. With the next oneAPI release the Linux wheels will build with a newer icx and still allow the old runtime. dpnp#3151 pins icx to 2026.2 and gets these from the compiler version through pin-compatible. Could we do the same here?
| @@ -1,18 +1,22 @@ | |||
| {% set version = "0.6.0" %} | |||
There was a problem hiding this comment.
The other repos take the version from GIT_DESCRIBE_TAG/GIT_DESCRIBE_NUMBER, and so did the old recipe. With 0.6.0 and build 0 hardcoded, every build from main looks the same as the 0.6.0 release. The "takes its version from git describe" comment in intel-ci.yml is also wrong now.
| --no-index \ | ||
| --prefix "${PREFIX}" \ | ||
| CC=icx CXX=icpx $PYTHON -m pip install --no-build-isolation --no-deps \ | ||
| -Csetup-args="-Dmkl_threading=gnu_thread" \ |
There was a problem hiding this comment.
The Linux conda package now uses gnu_thread with llvm-openmp, while the old recipe, Windows and the wheels use intel_thread with libiomp5. Is that intended? If these packages go to the Intel channel, it might be worth mentioning in the description.
| [tool.cython-lint] | ||
| ignore = ['E722'] # do not use bare 'except' | ||
| max-line-length = 80 | ||
| [tool.cibuildwheel] |
There was a problem hiding this comment.
[tool.cython-lint] got removed here. Was that on purpose? cython-lint still runs in pre-commit, now with its default settings.
| conda-remove-defaults: "true" | ||
|
|
||
| - name: Install conda-build | ||
| run: conda install -y -q -n base --override-channels -c conda-forge conda-build |
There was a problem hiding this comment.
mkl-service pins conda-build=26.9.1. Should we pin it here too?
| - {{ pin_compatible('mkl') }} | ||
| - {{ pin_compatible('intel-openmp') }} | ||
| - {{ pin_compatible('intel-cmplr-lib-rt') }} | ||
| # NumPy 1 vs 2 loops are chosen at compile time (USE_NUMPY_2), so a build |
There was a problem hiding this comment.
USE_NUMPY_2 was removed in gh-289, so this comment is out of date. Same for "CI overrides it with --numpy" in conda_build_config.yaml, since nothing passes --numpy now.
| wheel: "mkl_umath {platform} Wheels Python {python}" | ||
| test: | ||
| requires: [pytest] | ||
| benchmark: |
There was a problem hiding this comment.
intel-ci.yml doesn't run benchmark.py or read this section. Could ci/README.md mention that, like in dpnp#3151? The description also makes it sound like PRs run the benchmarks.
| - **Workflows:** `.github/workflows/` | ||
| - `conda-package.yml` — main conda build/test pipeline | ||
| - `conda-package-cf.yml` — conda-forge-oriented build/test pipeline | ||
| - `intel-ci.yml` — Intel CI: builds and tests the conda packages and the wheels; see `ci/README.md` |
There was a problem hiding this comment.
Nothing tests against the Intel channel anymore, but line 80 still says Intel's NumPy is required. Maybe update it and mention in the description that the Intel channel build is gone?
Adds
ci/and.github/workflows/intel-ci.yml, which build the conda packages and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment. The recipe skips macOS, asconda-forge/mkl_umath-feedstockdoes.conda-recipe/is now the only recipe: the conda-forge one that wasconda-recipe-cf/.conda-package.ymlandconda-package-cf.ymlare removed, asintel-ci.ymlbuilds and tests the recipe.python:list inconda_build_config.yaml. The wheels follow thepyproject.tomlclassifiers: 3.10 to 3.15 plus 3.14t and 3.15t.dpcpp_win-64, whichci/install_icx.pyinstalls. They leave MKL and the compiler runtime out.ci/build.pyadds their requirements to each wheel:mklfrom the mkl-devel version they were built with up to its next major, through[tool.gh-ci] pin-compatible, andintel-openmpandintel-cmplr-lib-rt>=2026.2,<2027, throughwheel-requires. icx links the Linux libraries with full RELRO, whichci/build.pychecks, along with a library search path inside$ORIGIN.meson.buildaddsLibraryto CMake's search path on Windows, where mkl-devel puts its CMake config.ci/test.pyrunspytest --pyargs mkl_umathon the installed package.mainand tags build them all.info/about.json.ci/benchmark.pyruns the asv suite inbenchmarks/against the installed package: one quick sample of each benchmark on a pull request, a full measurement on a tag.Miniforge is pinned to 26.7.2-0, as 26.9.4-0 makes
setup-miniconda'sconda initfail on Windows runners.Validation: run 42669879582 on
d7e1dc3builds and tests the reduced pull request matrix on Linux and Windows. Not run yet: the full matrix, which onlymainand tags build.