Add support for the free-threaded build (#178)

* Initial support for free threaded Python (#1)

* fix various errors and warnings seen on clang and gcc

* use manylinux 2_28 on x86_64 (this will be the default in manylinux 3)

* split CT_LAZY_FIELD_LIST into a new listing of ct_flags_mut flags

* Add CT_UNDER_CONSTRUCTION to indicate that a type is being mutated

* adjust assert

* only set CT_UNDER_CONSTRUCTION during lazy init

* Move CT_CUSTOM_FIELD_POS and CT_WITH_PACKED_CHANGE to ct_flags_mut

* rearrange and add asserts in do_realize_lazy_struct

* remove unhelpful comment

(Kumar will fix this soon!)

* replace ct_is_hidden with asserts

* clarify comment

* fix thread safety of _realize_c_struct_or_union

* fix missing unlock

* mark tests as thread unsafe

* mark TestZIntegration as thread unsafe

* fix headers to avoid duplicated definitions

* fix thread safety of enum (#10)

* fix thread safety of cffi (#13)

Co-authored-by: Nathan Goldbaum <[email protected]>

* add trove classifier

* add missing CI jobs

* back out unnecessary CI config changes

* pin cibuildwheel<3 in CI config

* reduce the number of pytest-run-parallel jobs

* remove unnecessary thread_unsafe markers, add reasons for all others

* expand whatsnew

* add 3.13t pytest-run-parallel jobs

* revert more unnecessary changes

* disable gil in extension produced by verify tests

* fail the tests if the GIL is enabled at tests completion

* force realizing lazy structs in ffi_sizeof

* remove out-of-date FIXME

* tweak error

* revert unnecessary change

* Unconditionally clear weak refs in ctypdescr_dealloc to avoid race in CPython internals.

see python/cpython#135607

* mark tests as thread-unsafe that might concurrently call dlclose on the same extension

* apply fix from c02343c in a few more spots

* check for lazy_field_list in force_lazy_struct

* use atomic loads for opcodes in realize_c_type_or_func_now

* address code review

* remove skips

* add thread safety note for realize_c_type_or_func

* use atomics for ct_stuff

* Simplify ct_stuff handling

* use _CFFI_LOAD_OP

* use one global cffi lock for thread safety

* do not export dummy

* skip

* unconditionally heap-allocate _ffi_type output

* move mutable flags to bitflags for ct_flags_mut field

* fix refleak

* Apply suggestions from code review

Co-authored-by: Kumar Aditya <[email protected]>

* apply matti's review comments

* update version numbers

* remove PYPY trove classifier

* expire pointer conversion deprecation to fix tests

* error on free-threaded Python 3.13

* drop 3.13t CI and wheel builds

* drop cibuilwheel ~=3.0b1

* delete now-unnecessary CIBW_ENABLE env var

---------

Co-authored-by: Sam Gross <[email protected]>
Co-authored-by: Kumar Aditya <[email protected]>

Author: 3 people

.github/workflows/ci.yaml

@@ -76,18 +76,18 @@ jobs:
           - { spec: cp310-manylinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp311-manylinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-manylinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp313-manylinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-manylinux_x86_64, arch: x86_64, cibw_version: cibuildwheel~=3.0b1 }
-          # FIXME: need to run tests with PYTHON_GIL=0 on this build to actually test sans-GIL, but breaks packaging tests that use the wrong `virtualenv` script wrapper 
-          - { spec: cp314t-manylinux_x86_64, skip_artifact_upload: 'true', cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp313-manylinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }          
+          - { spec: cp314-manylinux_x86_64, arch: x86_64 }
+          - { spec: cp314t-manylinux_x86_64, arch: x86_64 }
 
           # x86_64 musllinux
           - { spec: cp39-musllinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp310-musllinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp311-musllinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-musllinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-musllinux_x86_64, arch: x86_64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-musllinux_x86_64, arch: x86_64, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-musllinux_x86_64, arch: x86_64 }
+          - { spec: cp314t-musllinux_x86_64, arch: x86_64 }
 
           # i686 manylinux
           - { spec: cp39-manylinux_i686, arch: i686, omit: ${{ env.skip_ci_redundant_jobs }} }
@@ -109,31 +109,35 @@ jobs:
           - { spec: cp311-manylinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-manylinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-manylinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-manylinux_aarch64, arch: aarch64, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-manylinux_aarch64, arch: aarch64 }
+          - { spec: cp314t-manylinux_aarch64, arch: aarch64 }
 
             # aarch64 musllinux
           - { spec: cp39-musllinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp310-musllinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp311-musllinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-musllinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-musllinux_aarch64, arch: aarch64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-musllinux_aarch64, arch: aarch64, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-musllinux_aarch64, arch: aarch64 }
+          - { spec: cp314t-musllinux_aarch64, arch: aarch64 }
 
           # ppc64le manylinux
           - { spec: cp39-manylinux_ppc64le, arch: ppc64le, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs }} }
           - { spec: cp310-manylinux_ppc64le, arch: ppc64le, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp311-manylinux_ppc64le, arch: ppc64le, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp312-manylinux_ppc64le, arch: ppc64le, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp313-manylinux_ppc64le, arch: ppc64le, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-manylinux_ppc64le, arch: ppc64le, omit: ${{ env.skip_slow_jobs }}, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-manylinux_ppc64le, arch: ppc64le, omit: ${{ env.skip_slow_jobs }} }
+          - { spec: cp314t-manylinux_ppc64le, arch: ppc64le, omit: ${{ env.skip_slow_jobs }} }
 
           # s390x manylinux
           - { spec: cp39-manylinux_s390x, arch: s390x, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs }} }
           - { spec: cp310-manylinux_s390x, arch: s390x, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp311-manylinux_s390x, arch: s390x, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp312-manylinux_s390x, arch: s390x, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
           - { spec: cp313-manylinux_s390x, arch: s390x, test_args: '{package}/src/c', omit: ${{ env.skip_slow_jobs || env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-manylinux_s390x, arch: s390x, omit: ${{ env.skip_slow_jobs }}, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-manylinux_s390x, arch: s390x, omit: ${{ env.skip_slow_jobs }} }
+          - { spec: cp314t-manylinux_s390x, arch: s390x, omit: ${{ env.skip_slow_jobs }} }
 
   linux:
     needs: [python_sdist, make_linux_matrix]
@@ -178,7 +182,6 @@ jobs:
         CIBW_MUSLLINUX_X86_64_IMAGE: ${{ matrix.musllinux_img || 'musllinux_1_2' }}
         CIBW_MUSLLINUX_I686_IMAGE: ${{ matrix.musllinux_img || 'musllinux_1_2' }}
         CIBW_MUSLLINUX_AARCH64_IMAGE: ${{ matrix.musllinux_img || 'musllinux_1_2' }}
-        CIBW_ENABLE: cpython-prerelease cpython-freethreading
         CIBW_TEST_REQUIRES: pytest setuptools  # 3.12+ no longer includes distutils, just always ensure setuptools is present
         CIBW_TEST_COMMAND: PYTHONUNBUFFERED=1 python -m pytest ${{ matrix.test_args || '{project}' }}  # default to test all
       run: |
@@ -220,15 +223,17 @@ jobs:
           - { spec: cp311-macosx_x86_64, runs_on: [macos-13], omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-macosx_x86_64, runs_on: [macos-13], omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-macosx_x86_64, runs_on: [macos-13], omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-macosx_x86_64, runs_on: [macos-13], cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-macosx_x86_64, runs_on: [macos-13] }
+          - { spec: cp314t-macosx_x86_64, runs_on: [macos-13] }
 
           # arm64 macos
           - { spec: cp39-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp310-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', omit: ${{ env.skip_ci_redundant_jobs}} }
           - { spec: cp311-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}', cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}' }
+          - { spec: cp314t-macosx_arm64, deployment_target: '11.0', run_wrapper: 'arch -arm64 bash --noprofile --norc -eo pipefail {0}' }
 
   macos:
     needs: [python_sdist, make_macos_matrix]
@@ -263,7 +268,6 @@ jobs:
       id: build
       env:
         CIBW_BUILD: ${{ matrix.spec }}
-        CIBW_ENABLE: cpython-prerelease
         CIBW_TEST_REQUIRES: pytest setuptools
         CIBW_TEST_COMMAND: pip install pip --upgrade; cd {project}; PYTHONUNBUFFERED=1 pytest
         MACOSX_DEPLOYMENT_TARGET: ${{ matrix.deployment_target || '10.13' }}
@@ -306,21 +310,24 @@ jobs:
           - { spec: cp311-win_amd64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-win_amd64, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-win_amd64, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-win_amd64, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-win_amd64 }
+          - { spec: cp314t-win_amd64 }
           
           # x86 windows
           - { spec: cp39-win32, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp310-win32, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp311-win32, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp312-win32, omit: ${{ env.skip_ci_redundant_jobs }} }
           - { spec: cp313-win32, omit: ${{ env.skip_ci_redundant_jobs }} }
-          - { spec: cp314-win32, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp314-win32 }
+          - { spec: cp314t-win32 }
 
           # arm64 windows
-          - { spec: cp311-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }}, cibw_version: cibuildwheel~=3.0b1 }
-          - { spec: cp312-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }}, cibw_version: cibuildwheel~=3.0b1 }
-          - { spec: cp313-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }}, cibw_version: cibuildwheel~=3.0b1 }
-          - { spec: cp314-win_arm64, runs_on: windows-11-arm, cibw_version: cibuildwheel~=3.0b1 }
+          - { spec: cp311-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }} }
+          - { spec: cp312-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }} }
+          - { spec: cp313-win_arm64, runs_on: windows-11-arm, omit: ${{ env.skip_ci_redundant_jobs }} }
+          - { spec: cp314-win_arm64, runs_on: windows-11-arm }
+          - { spec: cp314t-win_arm64, runs_on: windows-11-arm }
 
   windows:
     needs: [python_sdist, make_windows_matrix]
@@ -345,7 +352,6 @@ jobs:
       id: build
       env:
         CIBW_BUILD: ${{ matrix.spec }}
-        CIBW_ENABLE: cpython-prerelease
         CIBW_TEST_REQUIRES: pytest setuptools
         CIBW_TEST_COMMAND: ${{ matrix.test_cmd || 'python -m pytest {package}/src/c' }}
         # FIXME: /testing takes ~45min on Windows and has some failures...
@@ -358,7 +364,7 @@ jobs:
         tar zxf cffi*.tar.gz --strip-components=1 -C cffi
 
         python -m pip install --upgrade pip
-        pip install "${{ matrix.cibw_version || 'cibuildwheel'}}"
+        pip install "${{ matrix.cibw_version || 'cibuildwheel' }}"
         python -m cibuildwheel --output-dir dist cffi
 
         echo "artifact_name=$(ls ./dist/)" >> "$GITHUB_OUTPUT"
@@ -384,10 +390,73 @@ jobs:
         delete-merged: true
       if: ${{ env.skip_artifact_upload != 'true' }}
 
+  make_run_parallel_matrix:
+    runs-on: ubuntu-24.04
+    outputs:
+      matrix_json: ${{ steps.make_matrix.outputs.matrix_json }}
+    steps:
+    - uses: actions/checkout@v4
+    - name: make a matrix
+      id: make_matrix
+      uses: ./.github/actions/dynamatrix
+      with:
+        matrix_yaml: |
+          include:
+          - { runner: ubuntu-latest, python-version: 3.14t-dev }
+          - { runner: macos-latest, python-version: 3.14t-dev }
+          - { runner: windows-latest, python-version: 3.14t-dev }
+
+
+  pytest-run-parallel:
+    needs: make_run_parallel_matrix
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJSON(needs.make_run_parallel_matrix.outputs.matrix_json) }}
+
+    runs-on: ${{ matrix.runner }}
+    steps:
+    - name: clone repo
+      uses: actions/checkout@v4
+
+    - name: install python
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: build and install
+      run: |
+        python -m pip install pytest setuptools pytest-run-parallel
+        python -m pip install .
+
+    - name: run tests under pytest-run-parallel
+      if: runner.os == 'Windows'
+      run: |
+        python -m pytest --parallel-threads=4 src/c
+
+    - name: run tests under pytest-run-parallel
+      if: runner.os != 'Windows'
+      run: |
+        python -m pytest --parallel-threads=4
+
+  clang_TSAN:
+    runs-on: ubuntu-latest
+    container: ghcr.io/nascheme/numpy-tsan:3.14t
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: build and install
+      run: |
+        python -m pip install setuptools pytest pytest-run-parallel
+        CFLAGS="-g -O3 -fsanitize=thread" python -m pip install -v .
+
+    - name: run tests under pytest-run-parallel
+      run: |
+        TSAN_OPTIONS="suppressions=$PWD/suppressions_free_threading.txt" \
+        python -m pytest --parallel-threads=4 --skip-thread-unsafe=True -sv
 
   check:
     if: always()
-    needs: [python_sdist, linux, macos, windows, merge_artifacts]
+    needs: [python_sdist, linux, macos, windows, clang_TSAN, pytest-run-parallel, merge_artifacts]
     runs-on: ubuntu-24.04
     steps:
     - name: Verify all previous jobs succeeded (provides a single check to sample for gating purposes)

doc/source/conf.py

@@ -45,9 +45,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '1.18'
+version = '2.0'
 # The full version, including alpha/beta/rc tags.
-release = '1.18.0.dev0'
+release = '2.0.0.dev0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/source/whatsnew.rst

@@ -2,11 +2,19 @@
 What's New
 ======================
 
-v1.18.0.dev0
+v2.0.0.dev0
 ============
 
+* Added support for free threaded Python. (`#178`_)
+  Note that the free-threaded build does not yet support building extensions
+  with the limited API, so you must set py_limited_api=False when building
+  extensions for the free-threaded build.
+* Added support for Python 3.14. (`#177`_)
 * WIP
 
+.. _`#177`: https://github.com/python-cffi/cffi/pull/177
+.. _`#178`: https://github.com/python-cffi/cffi/pull/178
+
 v1.17.1
 =======
 

pyproject.toml

@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "cffi"
-version = "1.18.0.dev0"
+version = "2.0.0.dev0"
 dependencies = [
     "pycparser; implementation_name != 'PyPy'",
 ]
@@ -26,8 +26,8 @@ classifiers = [
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
     "Programming Language :: Python :: 3.14",
+    "Programming Language :: Python :: Free Threading :: 2 - Beta",
     "Programming Language :: Python :: Implementation :: CPython",
-    "Programming Language :: Python :: Implementation :: PyPy",
 ]
 authors = [
     {name = "Armin Rigo"},

setup.py

@@ -1,4 +1,4 @@
-import sys, os, platform
+import sys, sysconfig, os, platform
 import subprocess
 import errno
 
@@ -16,6 +16,12 @@
 extra_compile_args = []
 extra_link_args = []
 
+FREE_THREADED_BUILD = bool(sysconfig.get_config_var('Py_GIL_DISABLED'))
+
+if FREE_THREADED_BUILD and sys.version_info < (3, 14):
+    raise RuntimeError("CFFI does not support the free-threaded build of CPython 3.13. "
+                       "Upgrade to free-threaded 3.14 or newer to use CFFI with the "
+                       "free-threaded build.")
 
 def _ask_pkg_config(resultlist, option, result_prefix='', sysroot=False):
     pkg_config = os.environ.get('PKG_CONFIG','pkg-config')