Wheel for Python 3.11 and x86_64, manylinux_2_17 becomes manylinux_2_28 - an issue?

[consideRatio]

During pip install pyzmq in a Centos 7 based docker image with Python 3.10 and Python 3.11 setup respectively, pip switches from downloading a wheel to downloading a source distribution. The reason is that for Python 3.11 of the x86_64 arch specifically, the published pyzmq wheel switched from manylinux_2_17 to 2_28.

Is this a constraint stemming from pyzmq, py311, or pyzmq's release automation as configured using cibuildwheel (recently #1707), or something else?

I tried to figure those details out myself but got stuck, so watch out - this issue can be off topic for pyzmq as a project.

Centos 7 with Python 3.10, pip install pyzmq

Collecting pyzmq>=17
  Downloading pyzmq-24.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)

Centos 7 with Python 3.11, pip install pyzmq

Collecting pyzmq>=17
  Downloading pyzmq-24.0.1.tar.gz (1.2 MB)

Related

• https://pypi.org/project/pyzmq/25.0.0b1/#files
• https://peps.python.org/pep-0599/
• https://peps.python.org/pep-0600/

[minrk]

Thanks for testing the beta! This is due to selecting the manylinux_2_28 image for cp311:

pyzmq/pyproject.toml

Lines 124 to 127 in 25545d4

	# manylinux_2_28 for cp311
	[[tool.cibuildwheel.overrides]]
	select = "cp311-*"
	manylinux-x86_64-image = "manylinux_2_28"

I thought it was reasonable to expect less than several-years-old OS with bleeding-edge Python (CentOS 7 is from 2014 and wildly outdated in almost all packages, but not quite at EOL, yet). There are benefits to using more recent glibc with libzmq. But this is partially an experiment! So if you've hit an issue in practice, I can use manylinux_2014 with cp311, or maybe even publish both.

Where did you encounter this?

[minrk]

I believe you'll also get a wheel if you use pip install --only-binary pyzmq pyzmq, which will ignore releases that don't have wheels for you, and downgrade pyzmq to the latest release with a wheel. That may become a reasonable strategy for folks sticking to extremely out-of-date software.

FWIW, glibc 2_17 is from 2012, while glibc 2.28 is only from 2018. So if you're on a glibc from less than 5 years ago you will get a pyzmq wheel with Python 3.11.

glibc >= 2.28 is available in:

• debian oldstable
• ubuntu 20.04
• fedora 38 (oldest supported)

[consideRatio]

Thanks Min for considering this and helping me understand what config influenced this!

I ran into this in the https://github.com/dask/dask-gateway 's CI system that has pre-built images providing: Slurm, OpenPBS, and Hadoop respectively - they are then used just in testing against those as backends to start dask clusters on.

These images have been setup and configured in Centos 7 historically, its tricky foe me to migrate as its a lot of know-how that i lack about the config even after trying to learn more for a long time. I was about to update to Centos 8 but come to understand that reached EOL before centos 7 or similar.

For this project, it seems bad to compromise performance or similar by building only for manylinux2.17 (aka. Manylinux2014), but if its a small detail to also build for manylimux 2.17 alongside without causing issues - that would be a notable 👍 for dask/dask-gateway.

[minrk]

I'll definitely look into the 'also'. I think cibuildwheel doesn't itself support a matrix of manylinux images, so I think I'll have to do it at the github workflow level, which might have a priority issue with the overrides section of the pyproject.toml configuration. Shouldn't be too hard, though.

The CentOS 8 / Stream situation is definitely a mess, and part of why folks are still on CentOS 7 (it even spawned whole new linux distros!). Upgrading the OS is often not a feasible option for folks, but compatibility with very-old OS and newer software is also not always an option, so being stuck on an old OS should expect to be a slowly growing inconvenience. Especially something as extremely outdated as CentOS (up to EOL next year). The --only-binary pyzmq option ought to work to prefer older wheels of pyzmq on an ever-supported OS.

I only wish there was a --prefer-binary option that was like --only-binary, but accepted source distributions of packages that don't have any wheels for you., so you could do e.g. --prefer-binary :all:

[consideRatio]

I only wish there was a --prefer-binary option that was like --only-binary, but accepted source distributions of packages that don't have any wheels for you., so you could do e.g. --prefer-binary :all:

Ah nice to understand that doing pip install --only-binary pyzmq before pip install -r requirements.txt would be a workaround to avoid the source distribution!

I considered how one could provide a manylinux2014 wheel alongside manylinux_2_28 wheels and ended up with this PR: #1823.

[minrk]

It doesn't need to be a separate command - pip install --only-binary pyzmq -r requirement.txt will work. The --only-binary pyzmq is telling pip to only consider binary candidates for pyzmq, it's not asking it to install pyzmq. So you can do e.g. pip install --only-binary pyzmq jupyterlab which is telling pip:

• please install jupyterlab
• if pyzmq is pulled in as a dependency, only consider binaries

The --only-binary (and --no-binary) flags are a bit surprising because they take arguments which are constraints but not requests to install. So they result in commands like:

pip install --only-binary pyzmq pyzmq
"only consider binary pyzmq" ^ | ^ "please install pyzmq"

[consideRatio]

@minrk it seems that pyzmq has never been built for python 3.11 with manylinux2014, so I could not use --only-binary=pyzmq. Btw the = part makes the flag easier to understand, as it could also be a flag that didn't accept a value - like --prefer-binary.

So that makes #1822 very appreciated!

+ pip install --only-binary=pyzmq pyzmq
ERROR: Could not find a version that satisfies the requirement pyzmq (from versions: none)
ERROR: No matching distribution found for pyzmq
Error: Process completed with exit code 1.

[consideRatio]

Aha, installing the conda-forge package worked still though - which was an option available to me in dask/dask-gateway's CI system.

[minrk]

@consideRatio Just checking, has pip been updated before running pip install, and what version of pip was used?