If you can say - are there any thoughts about implementing plugins / extension capabilities to keep type checking working even with libraries that aren't otherwise typecheckable?
(where "not otherwise typecheckable" means types that can't be expressed with stubs - e.g., Django, dataclasses pre-PEP-681, pytest fixtures, etc.)
At least for the moment, we aren't planning on a plugin architecture. We do recognize that there are some popular libraries and code patterns that aren't easily (or at all) typeable with the current state of the typing spec. We feel it would be more useful to help drive changes to the typing spec where we can, so that other type checkers can also benefit; and/or implement workarounds for the most popular libraries directly in ty, so that a library author doesn't have to rely on their downstream consumers installing a particular set of plugins to get good type-checker results.
(It's also more difficult to support plugins effectively in a type checker like ty, than in a linter like ruff, since any non-trivial use case would likely require deep changes to how we represent types and how we implement type inference. That's not something that lends itself to a couple of simple hook extension points.)
Helping improve the spec and all is great, but being 100% honest, as a user, I would rather have a type checker I can bend to my needs. As you said, some code patterns in a dynamic language like Python are difficult, or even impossible, to type-check without custom code. Type checkers are becoming more popular than ever, and this implicitly means that these code patterns are are going to be discouraged. On one hand, I believe the dynamism of Python is core to the language. On the other, I would never want to write any collaborative piece of software without a type checker anymore. Therefore, to get the benefits of a type checker, I am occasionally forced to write worse code just to please it.
Considering how fast uv and ruff took off, I am sure you are aware of the impact your project could have. I understand that supporting plugins is hard. However, if you are considering adding support for some popular libraries, IMHO, it would be really beneficial for the community if you could evaluate the feasibility of implementing things in a somewhat generic way, which could be then maybe leveraged by third-party authors.
Out of curiosity, do you have experience with other languages that have type system plugins that you’d hope be used as inspiration for something in Python?
I don’t have any such experience (short of a macro system, which requires code generation or runtime support) and it always makes me curious when people ask for type system plugins whether this is a standard feature in a type system I’ve never used.
To add to the complexity, you have to worry about not just which language you're analyzing, but also which language the type-checker is implemented in.
So if we were to do this for ty, we would have to carefully design the internal data types and algorithms that we use to model Python code, so that they're extensible in a robust way.
But we would also have to decide what kind of Rust plugin architecture to use. (Embed a Lua interpreter? dlopen plugins at runtime? Sidecar process communication over stdin/stdout?)
Solvable problems, to be sure, but it adds to the amount of work that's needed to support this well — which in turn affects our decisions about whether/when to prioritize this relative to other features.
Can you either give some additional details on the code patterns you’re talking about, or link to some ‘typical’ examples? I do appreciate the flexibility of being able to just write code and not particularly be overly sensitive to jumping through typing hoops, but I can’t think of any place I’ve actually used algorithms or specific code patterns that rely on untyped-ness to actually work at run time. I’d be very interested in trying to work through what is actually required to consider these code patterns as well-typed.
IMO creating custom rules is problematic - when projects import external code, rule conflicts become inevitable. C++'s type system might be complex, but at least there's consistency across header files within a project.
Regarding type checkers: while I don't love optimizing code just to make them run faster, most Python patterns can be implemented in statically checkable ways without much compromise. The benefits typically outweigh the costs. Python's dynamic features are powerful but rarely essential for everyday tasks.
https://pypi.org/project/django-types/ is compatible with pyright without plugins, so it should theoretically work with ty. It might not check as much as mypy though especially with values() querysets.
(where "not otherwise typecheckable" means types that can't be expressed with stubs - e.g., Django, dataclasses pre-PEP-681, pytest fixtures, etc.)