Use system-provided cyrus-sasl/libsasl2 at runtime

[jgiannuzzi]

Motivation

Currently the Python bindings require the user to build from source in order to use the SASL GSSAPI mechanism on Linux. This is because PyPI doesn't allow Python wheels that link to non-core libraries, like libsasl2.

Other bindings that can use the builds linking to libsasl2 - like the .NET ones - do not work on Debian-based systems without adding a symlink from libsasl2.so.3 to libsasl2.so.2. This is because the builds are done on an RPM-based system, which have a different soname ABI policy.

Implementation

Instead of having multiple builds with and without libsasl2 on Linux, we use dlopen to load libsasl2 at runtime on Unix. The SASL GSSAPI mechanism availability is thus checked at runtime.

The differences between Debian-based and RPM-based distros are addressed by probing the various known names for libsasl2. The subset of the ABI we use has not changed between the upstream soname ABI bumps.

The documentation has been updated to remove the previous limitations around libsasl2/GSSAPI.

The CI and build scripts have been updated to only build one flavor per linux/libc, and the --disable-gssapi parameter has been removed.

The 2 build systems (mklove and cmake) have been updated to build SASL GSSAPI support based on the availability of libdl instead of libsasl2.

The librdkafka.redist nuget package has been updated to include only one build for linux/glibc/x64, as the centos8-librdkafka.so build is obsoleted by the single build with no dependencies.

The static library build now supports the SASL GSSAPI mechanism.

A very small subset of the libsasl2 header has been included in rdkafka_sasl_cyrus.c. The license can be found at https://github.com/cyrusimap/cyrus-sasl/blob/master/COPYING.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ jgiannuzzi
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

Pull Request Overview

This PR implements runtime loading of the Cyrus SASL library (libsasl2) on Unix systems to enable SASL GSSAPI mechanism support without requiring static linking at build time. This change allows Python wheels to be distributed on PyPI while maintaining GSSAPI functionality and addresses ABI compatibility issues between different Linux distributions.

Key Changes

• Dynamic loading of libsasl2 at runtime using dlopen instead of build-time linking
• Removal of build dependencies on libsasl2 across all packaging configurations
• Simplification of build artifacts by eliminating separate GSSAPI and non-GSSAPI builds

Reviewed Changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/rdkafka_sasl_cyrus.c	Implements dynamic loading of libsasl2 with function pointer resolution and error handling
src/rdkafka_sasl_int.h	Adds new API functions for library status checking
src/rdkafka_sasl.c	Updates provider selection to check runtime library availability
src/rdkafka_conf.c	Updates error message for missing library support
src/CMakeLists.txt	Removes static linking to libsasl2
configure.self	Removes libsasl2 build dependency configuration
Various packaging files	Removes libsasl2 dependencies and GSSAPI build variants

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[jgiannuzzi]

Hi @emasab, could you please review this PR?