-
Notifications
You must be signed in to change notification settings - Fork 1.5k
[ty] Infer types for Callable
in the invalid arguments case
#19478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
ec9ba79
967d072
f10ffd3
231396c
2ba5a98
7222539
b60fcfa
7b155c0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -42,6 +42,7 @@ Or, when it's a literal type: | |
|
||
```py | ||
# error: [invalid-type-form] "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`" | ||
# error: [invalid-type-form] "Int literals are not allowed in this context in a type expression" | ||
def _(c: Callable[42, str]): | ||
reveal_type(c) # revealed: (...) -> Unknown | ||
``` | ||
|
@@ -58,8 +59,6 @@ def _(c: Callable[[int, 42, str, False], None]): | |
|
||
### Missing return type | ||
|
||
<!-- pull-types:skip --> | ||
|
||
Using a parameter list: | ||
|
||
```py | ||
|
@@ -84,12 +83,25 @@ Or something else that's invalid in a type expression generally: | |
# fmt: off | ||
|
||
def _(c: Callable[ # error: [invalid-type-form] "Special form `typing.Callable` expected exactly two arguments (parameter types and return type)" | ||
# error: [invalid-type-form] "Set literals are not allowed in type expressions" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Similarly here, I think the below error sort of covers this already? |
||
{1, 2} # error: [invalid-type-form] "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`" | ||
] | ||
): | ||
reveal_type(c) # revealed: (...) -> Unknown | ||
``` | ||
|
||
```py | ||
# fmt: off | ||
|
||
def _(c: Callable[ | ||
# error: [invalid-type-form] "Set literals are not allowed in type expressions" | ||
# error: [invalid-type-form] "Int literals are not allowed in this context in a type expression" | ||
{1, 2}, 2 # error: [invalid-type-form] "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`" | ||
] | ||
): | ||
reveal_type(c) # revealed: (...) -> Unknown | ||
``` | ||
|
||
### More than two arguments | ||
|
||
We can't reliably infer the callable type if there are more then 2 arguments because we don't know | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9601,18 +9601,37 @@ impl<'db> TypeInferenceBuilder<'db, '_> { | |
SpecialFormType::Callable => { | ||
let mut arguments = match arguments_slice { | ||
ast::Expr::Tuple(tuple) => Either::Left(tuple.iter()), | ||
_ => { | ||
self.infer_callable_parameter_types(arguments_slice); | ||
Either::Right(std::iter::empty::<&ast::Expr>()) | ||
} | ||
_ => Either::Right(std::iter::once(arguments_slice)), | ||
}; | ||
|
||
let first_argument = arguments.next(); | ||
|
||
let parameters = | ||
first_argument.and_then(|arg| self.infer_callable_parameter_types(arg)); | ||
// infer_callable_parameter_types(..) does not store the type, but in the case it is not a valid argument | ||
// we infer and store it. `storing_parameters_type_needed` avoids storing the `parameters` type multiple times. | ||
let parameters = first_argument.and_then(|arg| { | ||
let ty = self.infer_callable_parameter_types(arg); | ||
if ty.is_none() { | ||
self.infer_type_expression(arg); | ||
} | ||
Comment on lines
+9613
to
+9615
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This doesn't feel right (and it's the cause of the "unnecessary" extra diagnostics I mentioned in the tests). We know that whatever expression is here, it's not valid in this context. It doesn't make sense to then try to infer it as a type expression (when an arbitrary type expression is not valid here, even if its a valid type expression!). We are only doing this because it's the most convenient way to ensure a type is assigned to every sub-expression. If we had a "walk expression tree and store Unknown for every sub-expression" it would make more sense to use that here -- but we haven't built that utility. And really even that seems wasteful. I feel like this is pushing me over the edge where I am really ready to just implement a blanket fallback to Unknown (at query time) for any expression we didn't store a type for, and be done with the "no type stored for this expression" panics once and for all. I think it's fine if we occasionally miss an expression, end up with Unknown, and eventually get a bug report. And it can save us so much useless work (here I mean partially runtime work, but also development work) in error cases, where it would just become fine/correct to return early and not bother with storing types. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I totally understand your opinion on this ! I have learn quite a few tricks in rust playing around with this PR so it is ok if it doesn't get merge at all. I will be happy on working on the refactoring of "no type stored for this expression" if a new mechanism for handling such cases is introduced. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I tried that half a year ago when I was fed up with patching a lot of these panics :-)
Strong agree (see also my "side remark" in the linked PR). We could maybe still make it a hard error (or at least an error in debug mode) if we fall back to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I still have concerns about the impact this might have on the ability to use ty as a backend for Ruff in the long-term (#14629 (comment), #14629 (comment)). I agree that we've spent a frustrating amount of time chasing down corpus panics for invalid syntax and/or invalid type expressions. But these corpus tests have also caught some genuine bugs where we weren't storing subexpressions for nodes involving valid type expressions -- #18535 and #18536 were examples. I think it would be really confusing for users if they saw We are definitely too panicky in the server right now, though, and we definitely emit too many cascading diagnostics for invalid type expressions. My preferred solution for the first problem would be to keep the strict corpus tests, but to avoid indexing directly into the expression map from the server code -- instead we should do something like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I looked back at that prior discussion, and I am not convinced that future-type-aware-Ruff is sufficient motivation for us to work so hard to infer meaningful types for invalid expressions. Invalid expressions aren't common, issue their own errors, and tend to be fixed -- we don't need type-aware-Ruff to issue a cascading diagnostic for every other possible lint issue inside an invalid annotation. (I don't buy that there are any significant numbers of people who care about Ruff emitting a diagnostic for It's still not clear to me how you would actually propose to handle the case in this PR. Should we infer it as a type expression, even though it is not in a context where a type expression is either valid or expected? As a value expression, even though it is not a context where a value expression is valid or expected? Neither option makes semantic sense, and both risk unnecessary/confusing cascading errors. I don't know of any other option that's been suggested, other than "explicitly store Unknown for the whole tree." That option is significantly more work, and it still doesn't help the Ruff case. Yes, it's not ideal if we wrongly reveal I agree that at minimum we must start using a fallback when we pull types in the server, so we don't panic, but I don't think that's sufficient. If we maintain that the corpus tests must find an explicit type for every expression, that means we still (in principle, though we won't catch the bugs as quickly via playground/LSP panics) require ourselves to always find some way to infer and explicitly store a type for every expression, including in all error cases. And I think that is a bad use of our time, and we should stop doing it. I think the corpus tests should continue to exist, and should continue to pull a type for every expression, because that helps catch other failures/panics. But the corpus tests should also use a fallback to
Thank you for your understanding! I think we really already have most of the mechanism, and this is a relatively easy change to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I actually just went ahead and put up a PR for this, since it was quite simple -- though not quite as simple as I said above, due to #19435. I reused your added test from this PR, so I added you as a co-author. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'd still like to try out the idea I mentioned in astral-sh/ty#727 (comment), of refactoring |
||
ty | ||
}); | ||
let storing_parameters_type_needed = parameters.is_some(); | ||
|
||
let return_type = arguments.next().map(|arg| self.infer_type_expression(arg)); | ||
let return_type = match arguments.next() { | ||
Some(expr) => Some(self.infer_type_expression(expr)), | ||
None => { | ||
// In case we have a single argument, we return early since the type annotation is not valid. | ||
// We also make sure to store the `CallableType` to the `parameters` if `parameters` is valid. | ||
let ty = CallableType::unknown(db); | ||
if let Some(first_argument) = first_argument { | ||
if storing_parameters_type_needed { | ||
self.store_expression_type(first_argument, ty); | ||
} | ||
} | ||
report_invalid_arguments_to_callable(&self.context, subscript); | ||
return ty; | ||
} | ||
}; | ||
|
||
let correct_argument_number = if let Some(third_argument) = arguments.next() { | ||
self.infer_type_expression(third_argument); | ||
|
@@ -9637,10 +9656,13 @@ impl<'db> TypeInferenceBuilder<'db, '_> { | |
}; | ||
|
||
// `Signature` / `Parameters` are not a `Type` variant, so we're storing | ||
// the outer callable type on these expressions instead. | ||
// the outer callable type on these expressions instead, unless `first_argument` was invalid | ||
// for `Callable`, in which case a type is already stored. | ||
self.store_expression_type(arguments_slice, callable_type); | ||
if let Some(first_argument) = first_argument { | ||
self.store_expression_type(first_argument, callable_type); | ||
if storing_parameters_type_needed { | ||
self.store_expression_type(first_argument, callable_type); | ||
} | ||
} | ||
|
||
callable_type | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think ideally the first error here is sufficient and we don't need the second one, too. But if that's really difficult, it's not a big deal.