Unbreak CI (3.14 + cython) #3264

jakkdl · 2025-05-12T09:56:22Z

I'm for some reason not getting mypy fails locally, but disabling test_static_tool_sees_all_symbols entirely for 3.14.0b1 rather than trying to dig into why that might be the case

I'm also getting cffi/OpenSSL segfaults locally on 3.14, but that doesn't seem to impact CI so we can probs ignore that one for now

codecov · 2025-05-12T10:03:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.93728%. Comparing base (ac7b93f) to head (6ec7436).
⚠️ Report is 52 commits behind head on main.

Additional details and impacted files

@@                 Coverage Diff                 @@
##                main       #3264         +/-   ##
===================================================
+ Coverage   99.93727%   99.93728%   +0.00001%     
===================================================
  Files            126         126                 
  Lines          19129       19132          +3     
  Branches        1296        1296                 
===================================================
+ Hits           19117       19120          +3     
  Misses            10          10                 
  Partials           2           2

Files with missing lines	Coverage Δ
src/trio/_core/_tests/test_run.py	`100.00000% <100.00000%> (ø)`
src/trio/_tests/test_exports.py	`100.00000% <100.00000%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ci

jakkdl · 2025-05-12T11:24:49Z

an alternate fix for cython is to disable line traces, but that will disable coverage

jakkdl · 2025-05-12T11:31:08Z

ubuntu 3.14 now failed with a segfault I haven't seen in previous runs: https://github.com/python-trio/trio/actions/runs/14970947768/job/42051585863?pr=3264#step:6:1277

   Fatal Python error: Segmentation fault
  
  Thread 0x00007f1f7e4236c0 [Trio thread 0] (most recent call first):
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/site-packages/trio/_core/_thread_cache.py", line 194 in _work
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/threading.py", line 1021 in run
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/threading.py", line 1079 in _bootstrap_inner
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/threading.py", line 1041 in _bootstrap
  
  Current thread 0x00007f1f83bfeb80 [coverage] (most recent call first):
    Garbage-collecting
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/site-packages/trio/_core/_tests/tutil.py", line 59 in gc_collect_harder
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/site-packages/trio/_core/_tests/tutil.py", line 75 in ignore_coroutine_never_awaited_warnings
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/contextlib.py", line 148 in __exit__
    File "/opt/hostedtoolcache/Python/3.14.0-beta.1/x64/lib/python3.14/site-packages/trio/_core/_tests/test_run.py", line 933 in test_error_in_run_loop

src/trio/_tests/test_exports.py

A5rocks · 2025-05-12T12:26:12Z

src/trio/_tests/test_exports.py

 @slow
 # see comment on test_static_tool_sees_all_symbols
 @pytest.mark.redistributors_should_skip
+@pytest.mark.skipif(


jakkdl · 2025-05-12T13:15:15Z

ubuntu 3.14 now failed with a segfault I haven't seen in previous runs: https://github.com/python-trio/trio/actions/runs/14970947768/job/42051585863?pr=3264#step:6:1277

managed to minify the repro to

import trio
from contextlib import suppress
import gc

def test_error_in_run_loop() -> None:
    # Blow stuff up real good to check we at least get a TrioInternalError
    async def main() -> None:
        task = trio.lowlevel.current_task()
        task._schedule_points = "hello!"  # type: ignore
        await trio.lowlevel.checkpoint()
    with suppress(trio.TrioInternalError):
        trio.run(main)
    gc.collect()
    gc.collect() # removing this line makes the segfault disappear

but I only get it when running through pytest. Modifying it to directly run the function body without invoking pytest does not give any segfault.

It should be possible to minify it further and recreate the exceptions that are happening inside trio, and maybe some way of unrolling the pytest stack.

A5rocks · 2025-05-12T13:25:24Z

It would be a good idea to create a cpython issue for that just so they know it's a thing.

jakkdl · 2025-05-12T14:33:23Z

opened python/cpython#133932

…r_in_run_loop, add overeager strict xfail on export tests

jakkdl · 2025-05-12T14:56:36Z

There's a more proper fix for cython than pinning it: cython/cython#6865 (comment)

jakkdl · 2025-05-12T15:06:51Z

uh, did I somehow break CI or is github acting up?
edit: yeah it was me, turns out yaml doesn't like comments in that spot

A5rocks · 2025-05-12T16:33:52Z

.github/workflows/ci.yml

        (
          endsWith(matrix.python, '-dev')
          || endsWith(matrix.python, '-nightly')
+          || matrix.python == '3.14'


IMO it's cleaner to use 3.14-dev because it's implied that we should change it back once 3.14 stable is released.

that didn't work https://github.com/python-trio/trio/actions/runs/14970947768/job/42051585912

It still fails but then it's allowed to. (I think that run failed because the Ubuntu and MacOS failures, not the Windows one)

It still fails but then it's allowed to. (I think that run failed because the Ubuntu and MacOS failures, not the Windows one)

uh, this is the error:

Version 3.14-dev.0-alpha - 3.14-dev.X was not found in the local cache
Error: The version '3.14-dev.0-alpha - 3.14-dev.X' with architecture 'x86' was not found for Windows 2022.

Oops, I wasn't able to see the logs. We should probably remove that logic near setup-python(I think it's only there because legacy) but for now this new thing is fine too.

you're very right, I confused myself from some git blameing.

But regardless, let's just merge this as is to unblock other PRs and we can possibly clean this up later?

I personally don't like that allow-prereleases input of actions/setup-python and use 3.14-dev or more often ~3.14.0-0 to have it represent that it's not yet stable explicitly.

One day, I'll show you my reusable-tox.yml thing which will simplify this by a lot...

I think allow-prereleases is fine, since basically it saves some unnecessary churn (we should be pretty sure Trio will not need changes before switching 3.14-dev to 3.14) and allows marking some versions as allowed to fail just based on the version scheme.

Is there a trap to be aware of?

It's just that you're forced to keep updating the conditional on this line which is disconnected from the version string, even though they are tightly coupled through the concept of pre-releases. That's two places to keep in sync already with a visually unobvious link. This is what I consider mental overhead. The same churn just worse.

Having "allow-failures" in the same identifier allows being intentional about expecting a version to fail or not.

To be clear the only reason we had to update the conditional on this line is because some other legacy logic -- we don't actually have allow-prereleases but instead some text interpolation to emulate the effects. The downside of the text interpolation is that we can't support 3.14-dev, so we if switched to allow-prereleases we would have the same behavior as currently except that 3.14-dev would work.

And then since we have a continue-on-error we can use the version identifier to signal whether failure is allowed (3.14-dev will allow failure, 3.14 won't). This way we can actually make use of the fact that Python tries not to break anything after they start releasing betas (by switching to 3.14) while not having to switch the identifier whenever a stable version comes out.

src/trio/_core/_tests/test_run.py

src/trio/_tests/test_exports.py

follow up to #3264

skip static introspection on 3.14.0b1

8592150

jakkdl added 2 commits May 12, 2025 12:50

also skip sees_all_symbols, pin cython to <3.1.0

c36b62b

name 3.14 with -dev to pass if failing. Also restrict cython version in

361fd07

ci

A5rocks reviewed May 12, 2025

View reviewed changes

jakkdl changed the title ~~skip static introspection on 3.14.0b1~~ Unbreak CI (3.14 + cython) May 12, 2025

continue-on-error for windows3.14, add non-strict xfail for test_erro…

d9e62f1

…r_in_run_loop, add overeager strict xfail on export tests

switch back to skipif ... maybe unbreak CI?

768ca85

unbreak CI

20ff958

A5rocks reviewed May 12, 2025

View reviewed changes

jakkdl mentioned this pull request May 14, 2025

CPython 3.14 and Cython==3.1.0 CI issues #3267

Open

5 tasks

summer is finally here ☀

6ec7436

A5rocks approved these changes May 14, 2025

View reviewed changes

A5rocks merged commit bb63c53 into python-trio:main May 14, 2025
40 of 43 checks passed

jakkdl deleted the unbreak_314 branch May 15, 2025 10:53

jakkdl mentioned this pull request May 15, 2025

dont use matrix for conditional variables in CI aio-libs/aiobotocore#1357

Merged

jakkdl added a commit that referenced this pull request May 21, 2025

unpin cython (#3269)

ca218b2

follow up to #3264

jakkdl mentioned this pull request May 30, 2025

Push some local cleanups #3276

Merged

Uh oh!

Unbreak CI (3.14 + cython) #3264

Unbreak CI (3.14 + cython) #3264

Uh oh!

Conversation

jakkdl commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jakkdl commented May 12, 2025

Uh oh!

jakkdl commented May 12, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakkdl commented May 12, 2025

Uh oh!

A5rocks commented May 12, 2025

Uh oh!

jakkdl commented May 12, 2025

Uh oh!

jakkdl commented May 12, 2025

Uh oh!

jakkdl commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

A5rocks May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

A5rocks May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jakkdl commented May 12, 2025 •

edited

Loading

codecov bot commented May 12, 2025 •

edited

Loading

jakkdl commented May 12, 2025 •

edited

Loading

A5rocks May 14, 2025 •

edited

Loading

webknjaz May 27, 2025 •

edited

Loading

A5rocks May 30, 2025 •

edited

Loading