Refactor generated function implementation to provide bindings/method #57230

Keno · 2025-02-01T03:34:43Z

This PR refactors the generated function implementation in multiple ways:

Rather than allocating a new LineNumber node to pass to the generator, we just pass the original method from which this LineNumberNode was constructed. This has been a bit of a longer-standing annoyance of mine, since the generator needs to know properties of the original method to properly interpret the return value from the generator, but this information was only available on the C side.
Move the handling of Expr returns fully into Julia. Right not things were a bit split with the julia code post-processing an Expr return, but then handing it back to C for lowering. By moving it fully into Julia, we can keep the C-side interface simpler by always getting a CodeInfo.

With these refactorings done, amend the post-processing code to provide binding edges for Expr returns. Ordinarily, bindings in lowered code do not need edges, because we will scan the lowered code of the method to find them. However, generated functions are different, because we do not in general have the lowered code available. To still give them binding edges, we simply scan through the post-lowered code and all of the bindings we find into the edges array.

I will note that both of these will require minor adjustments to @generated functions that use the CodeInfo interface (N.B.: this interface is not considered stable and we've broken it in almost every release so far). In particular, the following adjustments need to be made:

Adjusting the source argument to the new Method ABI
If necessary, adding any edges that correspond to GlobalRefs used - the code will treat the returned CodeInfo mostly opaquely and (unlike in the Expr case) will not automatically compute these edges.

base/expr.jl

This PR refactors the generated function implementation in multiple ways: 1. Rather than allocating a new LineNumber node to pass to the generator, we just pass the original method from which this LineNumberNode was constructed. This has been a bit of a longer-standing annoyance of mine, since the generator needs to know properties of the original method to properly interpret the return value from the generator, but this information was only available on the C side. 2. Move the handling of `Expr` returns fully into Julia. Right not things were a bit split with the julia code post-processing an `Expr` return, but then handing it back to C for lowering. By moving it fully into Julia, we can keep the C-side interface simpler by always getting a `CodeInfo`. With these refactorings done, amend the post-processing code to provide binding edges for `Expr` returns. Ordinarily, bindings in lowered code do not need edges, because we will scan the lowered code of the method to find them. However, generated functions are different, because we do not in general have the lowered code available. To still give them binding edges, we simply scan through the post-lowered code and all of the bindings we find into the edges array. I will note that both of these will require minor adjustments to `@generated` functions that use the CodeInfo interface (N.B.: this interface is not considered stable and we've broken it in almost every release so far). In particular, the following adjustments need to be made: 1. Adjusting the `source` argument to the new `Method` ABI 2. If necessary, adding any edges that correspond to GlobalRefs used - the code will treat the returned CodeInfo mostly opaquely and (unlike in the `Expr` case) will not automatically compute these edges.

vtjnash · 2025-02-03T17:08:22Z

src/module.c

+JL_DLLEXPORT void jl_add_binding_backedge(jl_binding_t *b, jl_value_t *edge)
+{
+    if (!b->backedges) {
+        b->backedges = jl_alloc_vec_any(0);


It looks like there is no lock on this, so any uses of ./julia (which now defaults to using threads) and which run any code now may trigger this UB memory corruption?

vtjnash · 2025-02-03T17:14:14Z

base/expr.jl

+                         Expr(:meta, :pop_loc))))
+    spnames = g.spnames
+    return generated_body_to_codeinfo(spnames === Core.svec() ? lam : Expr(Symbol("with-static-parameters"), lam, spnames...),
+        typename(typeof(g.gen)).module,


Why aren't we using source.module here anymore? I suppose they probably usually resolve to the same thing, but might be good to document the intent of this change anyways?

This addresses post-commit review #57230 (comment). This change was left-over from before I decided to also change the type of the `source` argument (at which point `source.module` was unavailable in the function). This module was supposed to be the same, but it turns out that both the julia tests and several packages use this code manually and use different modules for the two places. Use the same one we used before (which is probably more correct anyway) to fix #57417

This addresses post-commit review #57230 (comment). This change was left-over from before I decided to also change the type of the `source` argument (at which point `source.module` was unavailable in the function). This module was supposed to be the same, but it turns out that both the julia tests and several packages use this code manually and use different modules for the two places. Use the same one we used before (which is probably more correct anyway) to fix #57417 (cherry picked from commit 0c5372f)

KristofferC · 2025-02-27T10:24:20Z

Seems like this broke Cassette?

https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/c78016e_vs_d63aded/BorrowChecker.primary.log
https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/c78016e_vs_d63aded/CountFlops.primary.log

etc

Keno · 2025-02-28T04:45:11Z

Cassette has several issues on 1.12+ (e.g. JuliaLabs/Cassette.jl@7e798c2), but yes, it may need a tweak for this as well.

* Adapt to JuliaLang/julia#56509 * Adapt to JuliaLang/julia#54734 * Use StmtRange explicitly * Adapt to JuliaLang/julia#57230 * Reuse Cthulhu code structure for Compiler cache/finish overrides * Adapt to JuliaLang/julia#57475 * Adapt to JuliaLang/julia#55976 * Adapt to JuliaLang/julia#54734 * Use CC instead of .Compiler * Implement ir.argtypes[1] fix from JuliaLang/julia#54458 * Comment out failing tests To highlight which are broken, should probably be fixed before merging * Treat `getproperty(::Module, ::Symbol)` like GlobalRefs * Uncomment passing tests, explicitly mark others as broken * Evaluate GlobalRef only if binding is defined * Use `rrule` for getproperty(::Module, ::Symbol) * Bump compat bound for StructArrays * Raise compat bound for Cthulhu * Revert `isconst` change now that it is fixed * Adapt to `finishinfer!` signature change --------- Co-authored-by: Cédric Belmant <[email protected]>

…efined) Fixes a discrepancy between the code in C before #57230 and in Julia afterwards, making sure to sequence these definitions correctly. Not sure how to write a reliable test since it is specific to when this generated function is defined relative to the helpers used by this thunk, but the issue/fix is visible with: ``` $ ./julia -e 'code_lowered(ntuple, (Returns{Nothing}, Val{1000000}))' ```

Adapts to JuliaLang/julia#57230

…efined) Fixes a discrepancy between the code in C before #57230 and in Julia afterwards, making sure to sequence these definitions correctly. Not sure how to write a reliable test since it is specific to when this generated function is defined relative to the helpers used by this thunk, but the issue/fix is visible with: ``` $ ./julia -e 'code_lowered(ntuple, (Returns{Nothing}, Val{1000000}))' ```

… defined) (#57925) Fixes a discrepancy between the code in C before #57230 and in Julia afterwards, making sure to sequence these method definitions correctly. Not sure how to write a reliable test since it is specific to when this generated function is defined relative to the helpers used by this thunk, but the issue/fix is visible with: ``` $ ./julia -e 'code_lowered(ntuple, (Returns{Nothing}, Val{1000000}))' ``` Fix #57301

… defined) (#57925) Fixes a discrepancy between the code in C before #57230 and in Julia afterwards, making sure to sequence these method definitions correctly. Not sure how to write a reliable test since it is specific to when this generated function is defined relative to the helpers used by this thunk, but the issue/fix is visible with: ``` $ ./julia -e 'code_lowered(ntuple, (Returns{Nothing}, Val{1000000}))' ``` Fix #57301 (cherry picked from commit a3c48d7)

Adapts to JuliaLang/julia#57230

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

@aviatesk

* Update CodeInfo struct and handling Co-authored-by: Claire Foster <[email protected]> * Don't produce raw symbol from globalref This used to implicitly refer to a module-level name, but lowering is now expected to wrap it in a `globalref`. Part of JuliaLang/julia#54772 * Updates to const and global lowering; add K"constdecl"; omit `wrap` JuliaLang/julia#54773, JuliaLang/julia#56713, JuliaLang/julia#57470. Some changes omitted from `expand-decls` and `expand-assignment`. Note that the two-argument IR "const" is K"constdecl", whereas the one-argument K"const" only appears in the AST. Also note that the `wrap` parameter is omitted throughout assignment desugaring. As far as I'm aware, all this plumbing was just to support `const a,b,c = 1,2,3` having `b` and `c` inherit the `const`. TODO: find a better way of doing the same thing (a ScopedValue might be a clean solution; we currently throw an error). The check for `let; const x = 1; end`, (which should throw) is in scope analysis (lisp has it in `compile`). Co-authored-by: Claire Foster <[email protected]> * Add `isdefinedglobal` builtin JuliaLang/julia#54999, JuliaLang/julia#56985 * :global no longer valid_ir_argument; rm `is_defined_nothrow_global` JuliaLang/julia#56746. Also call :slot and :static_parameter valid (for now) * Fix `is_defined_and_owned_global` (Core.Binding changes) Adapt to bpart changes in JuliaLang/julia#54788 * Struct desugaring: "Undo decision to publish incomplete types..." JuliaLang/julia#56497; Add self-referencing struct shim I have doubts about how long this solution will stay in the base repository, and how complete it is (doesn't seem to work with M1.M2.S), but we are testing for it here. Also change the expected value of a test changed in the same PR. * Emit `latestworld` world age increments For method defs, `latestworld` is produced in desugaring rather than closure conversion for now (our closure conversion doesn't seem to cover the same cases as lisp lowering yet). Covers JuliaLang/julia#56523, JuliaLang/julia#56509, JuliaLang/julia#57299. Also includes changes from JuliaLang/julia#57102 (bpart: Start enforcing minimum world age for const bparts) and JuliaLang/julia#57150 (bpart: Start enforcing min_world for global variable definitions) since the lowering changes from those appear to be amendments to the changes above (missing world age increments). Co-authored-by: Claire Foster <[email protected]> * bpart changes: `Core._typebody!` signature `Core._typebody!` now takes a new "prev" argument, which we don't use yet here. Changes from JuliaLang/julia#57253 * bpart changes: struct desugaring Changes from JuliaLang/julia#57253 (bpart: Fully switch to partitioned semantics). This fixes one failing test and realigns struct desugaring to match lisp for now. Also changed: the expected result of redefining a primitive type (now allowed). * Additional argument in `new_opaque_closure` Fix segfaulting test. Thanks for the TODO * Adapt to different `GeneratedFunctionStub` signature Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help! * Fix `public` and `export` As of JuliaLang/julia#57765, `jl_module_public` is no longer exported. Change our runtime to handle it like `public` and `export` like we handle `import` or `using` for now * Fix modules.jl test I believe this was a world age issue * Regenerate IR tests Too many to count. * Update README to known-good julia, JuliaSyntax versions Latest julia works. Changes are needed to work with the latest JuliaSyntax, but that isn't in base julia yet, and more changes are likely to come. * Fix small bug from #16 so tests pass The change lifted the scope of `note`, so it was being changed in the loop * Changes from code review: const/global lowering Ping me if you'd like this squashed into the original const/global commit! Co-authored-by: Claire Foster <[email protected]> * Remove a special case No longer needed since we no longer put `global` or `local` forms back into the expand_forms machine. Some error messages change slightly as a result. * Changes from code review Co-authored-by: Claire Foster <[email protected]> * Fix + test for assignment in value but not tail position * Disallow `static_parameter` as `valid_ir_argument` See added comment, and discussion at #10 (comment) Co-authored-by: Claire Foster <[email protected]> * Change printing of `K"latestworld"` Parens are nice, but it wasn't consistent. Also make it a leaf (remaining non-leaves are deleted in the next commit.) * Move most `latestworld`s to linearization From the docs: ``` The following statements raise the current world age: 1. An explicit invocation of Core.@latestworld 2. The start of every top-level statement 3. The start of every REPL prompt 4. Any type or struct definition 5. Any method definition 6. Any constant declaration 7. Any global variable declaration (but not a global variable assignment) 8. Any using, import, export or public statement 9. Certain other macros like eval (depends on the macro implementation) ``` This commit handles each case as follows: ``` 1. = 9 2. I'm not sure this actually happens (or needs to happen, unless we're being defensive? Doing it after each world-changing operation should suffice). But if we need it, this would just be emitting once at the beginning of every lowered output. 3. = 2 4. = 6 5. Emit seeing `method` in linearize 6. Emit seeing `constdecl` in linearize 7. Emit seeing `global` or `globaldecl` in linearize 8. We just defer to `eval`, but should probably go in desugaring later - using/import recently became builtin calls, and I haven't updated JL to use them yet. Base._import_using has an expr-based API that may change, and our importpath destructuring is worth keeping. - export and public (special forms) are handled in toplevel.c 9. Done for us ``` Other quirks: - `JuliaLowering.eval_closure_type` calls eval to assign a const, so we still need to deal with that in closure conversion. - The `include` hack isn't mentioned in the docs, but can stay in desugaring. I'm not certain why we don't do the same for non-macro `eval`. --------- Co-authored-by: Claire Foster <[email protected]>

Keno mentioned this pull request Feb 1, 2025

WIP: World-age partition bindings #54654

Closed

14 tasks

Keno force-pushed the kf/generatedrefactor branch 3 times, most recently from 018796a to a016b2b Compare February 2, 2025 02:04

aviatesk reviewed Feb 2, 2025

View reviewed changes

base/expr.jl Outdated Show resolved Hide resolved

Keno force-pushed the kf/generatedrefactor branch 6 times, most recently from 3fb8912 to 62b3fcf Compare February 2, 2025 23:26

Keno force-pushed the kf/generatedrefactor branch from 62b3fcf to 30c6ff9 Compare February 2, 2025 23:30

Keno merged commit ce507a7 into master Feb 3, 2025
5 of 7 checks passed

Keno deleted the kf/generatedrefactor branch February 3, 2025 02:47

vtjnash reviewed Feb 3, 2025

View reviewed changes

maleadt mentioned this pull request Feb 10, 2025

bpart-related IR verification regression #57329

Closed

Keno mentioned this pull request Feb 14, 2025

generated: Switch resolution module back to what it was before #57419

Merged

serenity4 added a commit to serenity4/Diffractor.jl that referenced this pull request Feb 21, 2025

Adapt to JuliaLang/julia#57230

524ac00

vtjnash mentioned this pull request Mar 28, 2025

(spurious?) global binding world age warning involving function that returns a huge tuple #57301

Closed

vtjnash mentioned this pull request Mar 28, 2025

fix generated_body_to_codeinfo to avoid string (which is not always defined) #57925

Merged

timholy added a commit to JuliaDebug/JuliaInterpreter.jl that referenced this pull request Apr 1, 2025

Update generated functions for Julia 1.12

0212df1

Adapts to JuliaLang/julia#57230

timholy mentioned this pull request Apr 1, 2025

Update generated functions for Julia 1.12 JuliaDebug/JuliaInterpreter.jl#668

Merged

timholy added a commit to JuliaDebug/JuliaInterpreter.jl that referenced this pull request Apr 5, 2025

Update generated functions for Julia 1.12 (#668)

b552e60

Adapts to JuliaLang/julia#57230

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Apr 8, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

4aeea82

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Apr 18, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

f82b490

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

aviatesk pushed a commit to aviatesk/JuliaLowering.jl that referenced this pull request Jun 14, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

aa4b2ae

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

aviatesk pushed a commit to aviatesk/JuliaLowering.jl that referenced this pull request Jun 14, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

13ef72c

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

aviatesk pushed a commit to aviatesk/JuliaLowering.jl that referenced this pull request Jul 2, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

d45242b

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

aviatesk pushed a commit to aviatesk/JuliaLowering.jl that referenced this pull request Jul 2, 2025

Adapt to different GeneratedFunctionStub signature, add a worldinc

6c849fc

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Jul 29, 2025

Adapt to different GeneratedFunctionStub signature

bac1331

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Jul 29, 2025

Adapt to different GeneratedFunctionStub signature

1d78605

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Jul 30, 2025

Adapt to different GeneratedFunctionStub signature

0dd6c87

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

mlechu added a commit to mlechu/JuliaLowering.jl that referenced this pull request Aug 2, 2025

Adapt to different GeneratedFunctionStub signature

039627c

Signature changed in JuliaLang/julia#57230. Thanks @aviatesk for the help!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Refactor generated function implementation to provide bindings/method #57230

Refactor generated function implementation to provide bindings/method #57230

Uh oh!

Keno commented Feb 1, 2025

Uh oh!

Uh oh!

Uh oh!

vtjnash Feb 3, 2025 •

edited

Loading

Uh oh!

vtjnash Feb 3, 2025

Uh oh!

KristofferC commented Feb 27, 2025

Uh oh!

Keno commented Feb 28, 2025

Uh oh!

Uh oh!

Uh oh!

Refactor generated function implementation to provide bindings/method #57230

Refactor generated function implementation to provide bindings/method #57230

Uh oh!

Conversation

Keno commented Feb 1, 2025

Uh oh!

Uh oh!

Uh oh!

vtjnash Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vtjnash Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

KristofferC commented Feb 27, 2025

Uh oh!

Keno commented Feb 28, 2025

Uh oh!

Uh oh!

vtjnash Feb 3, 2025 •

edited

Loading