[ty] Implement stdlib stub mapping #19529

Gankra · 2025-07-24T13:18:14Z

by using essentially the same logic for system site-packages, on the assumption that system site-packages are always a subdir of the stdlib we were looking for.

This requires some more testing, but at least on my macbook it lets me jump to e.g. warnings.deprecated.

github-actions · 2025-07-24T13:22:02Z

`mypy_primer` results

No ecosystem changes detected ✅
No memory usage changes detected ✅

Gankra · 2025-07-24T13:23:23Z

crates/ty_project/src/metadata/options.rs

+        let real_stdlib_path = python_environment.as_ref().and_then(|python_environment| {
+            // For now this is considered non-fatal, we don't Need this for anything.
+            python_environment.real_stdlib_path(system).ok()
+        });
+


As implemented this will redo a lot of work the site-packages path does -- for now I'm leaving them separate in case the implementations should diverge more, but I can merge them together once we're more confident.

Gankra · 2025-07-24T13:24:06Z

crates/ty_python_semantic/src/module_resolver/resolver.rs

+    // Append the real stdlib as the very last option in search.
+    // Normally this means it will always be shadowed by typeshed.
+    //
+    // FIXME(Gankra): ideally this should be completely disabled unless we're in
+    // `ModuleResolveMode::NoStubsAllowed`.
+    if let Some(real_stdlib_path) = real_stdlib_path {
+        dynamic_paths.push(real_stdlib_path.clone());
+    }


Need to iterate on this part.

crates/ty_python_semantic/src/module_resolver/resolver.rs

Gankra · 2025-07-24T13:27:46Z

crates/ty_python_semantic/src/site_packages.rs

+            // FIXME(Gankra): should we inherit info from the parent environment?
+            parent_environment: _,


I don't think we should be using this? But I'm not sure.

I think base_executable_home_path should always be the same for the parent environment? It would be pretty weird if not? But you're the uv expert here ;)

Gankra · 2025-07-24T13:28:27Z

crates/ty_test/src/lib.rs

+            // FIXME(Gankra): should this be computed properly?
+            real_stdlib_path: None,


(Note: this is in testing code)

AlexWaygood · 2025-07-24T13:28:29Z

crates/ty_python_semantic/src/module_resolver/resolver.rs

+    // Append the real stdlib as the very last option in search.
+    // Normally this means it will always be shadowed by typeshed.
+    //
+    // FIXME(Gankra): ideally this should be completely disabled unless we're in
+    // `ModuleResolveMode::NoStubsAllowed`.
+    if let Some(real_stdlib_path) = real_stdlib_path {
+        dynamic_paths.push(real_stdlib_path.clone());
+    }


hmm, I'm not sure it's correct for this to come after e.g. the site-packages search path. I think this would mean that if e.g. the user had the moribund asyncio package from PyPI installed into their venv, they had import asyncio in a file, and they did go-to-definition on the asyncio symbol, we'd jump to the module in the venv's site-packages directory. But at runtime, the module in the venv's site-packages directory will be totally ignored: the stdlib takes precedence in module resolution over third-party packages, so import asyncio will resolve to the stdlib asyncio module.

oh sorry, I posted this comment before I saw your comment above

No this is still a great insight, thanks!

AlexWaygood · 2025-07-24T13:37:26Z

Here's the things I'd want to manually double-check that this works with:

A uv-managed Python installation
A Brew-installed Python installation
A pyenv installation of Python
GraalPy
PyPy

(If it doesn't work with all of them right now, obviously fine to TODO some of them and leave them for followups!)

Gankra · 2025-08-07T14:55:53Z

Hmm ok so this works great with uv-managed cpython/pypy/graalpy but need to work harder for homebrew at least. It looks like they go out of their way to make site-packages follow this format, but the stdlib is somewhere else. Notably the python binary it refers to is symlinked, and if we follow that symlink then this logic once again works (the stdlib is in fact in ../Frameworks/Python.framework/Versions/3.13/lib/python3.13/).

by using essentially the same logic for system site-packages, on the assumption that system site-packages are always a subdir of the stdlib we were looking for.

Gankra · 2025-08-07T16:35:18Z

Newest commit handles homebrew and the macos system python.

github-actions · 2025-08-07T16:36:24Z

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

MichaReiser

Nice!

Would it be very involved to write a test for this similar to the other tests we have in site_packages?

crates/ty_project/src/metadata/options.rs

MichaReiser · 2025-08-08T08:11:24Z

crates/ty_python_semantic/src/module_resolver/resolver.rs

+        let stdlib_path = if stdlib_path_is_shadowed {
+            None
+        } else {
+            Some(stdlib_path)
+        };
+        let real_stdlib_path = if real_stdlib_path_is_shadowed {
+            None
+        } else {
+            real_stdlib_path
+        };


Curious to hear @AlexWaygood's thoughts on this. I suspect that ty's results will be fairly bonkers when the standard library is shadowed

They mentioned astral-sh/ty#523

MichaReiser · 2025-08-08T08:16:38Z

crates/ty_python_semantic/src/site_packages.rs

+        if matches!(implementation, PythonImplementation::CPython)
+            && python_version.is_some_and(PythonVersion::free_threaded_build_available)
+        {
+            let alternative_path =
+                sys_prefix_path.join(format!("lib/python{}t", python_version.unwrap()));
+            if system.is_directory(&alternative_path) {
+                return Ok(alternative_path);
+            }
+        }
+    }


Nit: If a small preference to use a match or let here for python_version to avoid the unwrap. It's, unfortunately, a bit more verbose without if let chains (only one more month before we can use them)

Suggested change

if matches!(implementation, PythonImplementation::CPython)

&& python_version.is_some_and(PythonVersion::free_threaded_build_available)

{

let alternative_path =

sys_prefix_path.join(format!("lib/python{}t", python_version.unwrap()));

if system.is_directory(&alternative_path) {

return Ok(alternative_path);

}

}

}

match (implementation, python_version) {

(PythonImplementation::CPython, Some(python_version)) if python_version.free_threaded_build_available() => {

let alternative_path =

sys_prefix_path.join(format!("lib/python{}t", python_version.unwrap()));

if system.is_directory(&alternative_path) {

return Ok(alternative_path);

}

}

}

if matches!(implementation, PythonImplementation::CPython)

&& python_version.is_some_and(PythonVersion::free_threaded_build_available)

{

}

}

note that this is copy-pasted logic from the equivalent site-packages logic (prioritized keeping them the same where i can)

crates/ty_python_semantic/src/site_packages.rs

Gankra · 2025-08-08T19:51:52Z

I am going to land this, I'm reasonably confident in how safe the implementation is. I expect more bugs will come up with us failing to find Fun stdlibs but this is still a big win.

* main: (31 commits) Add AIR301 rule (#17707) Avoid underflow in default ranges before a BOM (#19839) Update actions/download-artifact digest to de96f46 (#19852) Update docker/login-action action to v3.5.0 (#19860) Update rui314/setup-mold digest to 7344740 (#19853) Update cargo-bins/cargo-binstall action to v1.14.4 (#19855) Update actions/cache action to v4.2.4 (#19854) Update Rust crate hashbrown to v0.15.5 (#19858) Update Rust crate camino to v1.1.11 (#19857) Update Rust crate proc-macro2 to v1.0.96 (#19859) Update dependency ruff to v0.12.8 (#19856) SIM905: Fix handling of U+001C..U+001F whitespace (#19849) RUF064: offer a safe fix for multi-digit zeros (#19847) Clean up unused rendering code in `ruff_linter` (#19832) [ty] Add Salsa caching to `TupleType::to_class_type` (#19840) [ty] Handle cycles when finding implicit attributes (#19833) [ty] fix goto-definition on imports (#19834) [ty] Implement stdlib stub mapping (#19529) [`flake8-comprehensions`] Fix false positive for `C420` with attribute, subscript, or slice assignment targets (#19513) [ty] Implement module-level `__getattr__` support (#19791) ...

Gankra requested review from carljm, AlexWaygood, sharkdp, dcreager and MichaReiser as code owners July 24, 2025 13:18

Gankra commented Jul 24, 2025

View reviewed changes

crates/ty_python_semantic/src/module_resolver/resolver.rs Outdated Show resolved Hide resolved

Gankra commented Jul 24, 2025

View reviewed changes

AlexWaygood reviewed Jul 24, 2025

View reviewed changes

ntBre added the ty Multi-file analysis & type inference label Jul 24, 2025

AlexWaygood added the server Related to the LSP server label Jul 24, 2025

carljm removed their request for review July 25, 2025 01:12

AlexWaygood mentioned this pull request Jul 29, 2025

[ty] Add a flow diagram for Python's import resolution algorithm #19620

Merged

Gankra force-pushed the gankra/sysroot branch from a0d64b0 to 9712de0 Compare August 7, 2025 14:29

Gankra added 4 commits August 7, 2025 12:32

[ty] Implement stdlib stub mapping

bcee961

by using essentially the same logic for system site-packages, on the assumption that system site-packages are always a subdir of the stdlib we were looking for.

fixup

7403f35

More rigorously handle real stdlibs

4790510

handle macos facades

895e040

Gankra force-pushed the gankra/sysroot branch from 1765c20 to 895e040 Compare August 7, 2025 16:34

MichaReiser approved these changes Aug 8, 2025

View reviewed changes

Gankra added 2 commits August 8, 2025 10:05

review fixups

fbd38e5

test real stdlib logic

d0daf7e

Gankra merged commit 7cc3f1e into main Aug 8, 2025
38 checks passed

Gankra deleted the gankra/sysroot branch August 8, 2025 19:52

		// FIXME(Gankra): should we inherit info from the parent environment?
		parent_environment: _,

		// FIXME(Gankra): should this be computed properly?
		real_stdlib_path: None,

[ty] Implement stdlib stub mapping #19529

[ty] Implement stdlib stub mapping #19529

Uh oh!

Conversation

Gankra commented Jul 24, 2025

Uh oh!

github-actions bot commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexWaygood commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gankra commented Aug 7, 2025

Uh oh!

Gankra commented Aug 7, 2025

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gankra Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Gankra commented Aug 8, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 24, 2025 •

edited

Loading

`mypy_primer` results

AlexWaygood commented Jul 24, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

Gankra Aug 8, 2025 •

edited

Loading