One reason was so the type system could be backported. You can install the typing module all the way back to Python 2.7, where list[int] will never work but nothing prohibits typing.List[int].
The system has been rolled out slowly in general, without hasty changes to the core language.
I like adding | as an alias for Union, and the PEP has lots of examples of where typing.Union gets gnarly really quick, but that doesn't seem like a great example in the changelog:
def square(number: int | float) -> int | float:
return number ** 2
A TypeVar constrained to int and float, T = TypeVar('T', int, float), should be used instead to indicate that the return type is the same as the argument type, no?
This example demonstrates why I don't understand Python's type annotations. They seem to be incompatible with the language's core principle of duck typing.
Why would you constrain a function like `square` to only accept `int`s and `float`s? What if I want to square, say, a `fractions.Fraction`?
Interface typing > duck typing. With any sort of code completion, it's 1000x easier and more productive to figure out what goes where when the computer has some idea of the types of a given varaible, and hence what operations you can do to it.
It's like duck typing with a built-in Audubon society taxonomy chart.
You can always do whatever you want at runtime, it's still python.
IMHO, duck typing is strictly inferior to interface typing. You'd define a `Number` interface which `int`, `float`, ect all implement.
Duck typing has been fundamental to Python for a long time. Do you think we're seeing a shift away from it? In the future, do you think "Pythonic" code will include significant use of explicit interfaces?
> Duck typing has been fundamental to Python for a long time. Do you think we're seeing a shift away from it?
We have been since at least the introduction of abstract base classes; even in purely dynamic python, having the ability to more explicitly declare and interrogate intent than pure duck typing is frequently useful.
> do you think "Pythonic" code will include significant use of explicit interfaces?
Sure, as it already does via abcs. But it will also still use lots of duck typing, though over time more of it will be “static duck typing” by way of protocols.
Are "protocols" the python term for what Go (and I guess java but I'm way rusty at java) calls "interfaces"? eg the set of methods exposed by an object.
>>> import numpy as np
>>> np.array("asdgh")**2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: ufunc 'square' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
There is a numbers.Number for that - but yeah, that's another way in which it's just not a good example..
I think typing_extensions.Protocol helps support the general duck typing "any object that has method x", though there are some implementation issues w/ dataclasses and such..
That won't help if I want to square a matrix, or a polynomial.
> typing_extensions.Protocol helps support the general duck typing "any object that has method x"
Even if there's a `powerable` protocol that could be used to annotate `square` and `cube`, how would you annotate a slightly more complex function like `poly`?
> how is a user to find out whether square really accepts a matrix?
How do you find out that a function exists, what it accepts, and what it does normally?
> From comments? They are again limited by the authors imagination.
Sure, but an author which much experience with duck typing will typically write documentation that specified that expected methods (informal structural “types”) rather than nominal types.
With the static equivalent (protocols), it’s conceivable that this is also discoverable via tooling.
Using maths to either oppose or support generic duck typing is just pointless, because it's easy to come up with counterexamples where a mathematical operation would or would not behave as expected.
In quite a few cases where one would overload a mathematical operation to, say, sum two objects of certain type, it'd be better to explicitly type out the operation as a functor/lambda.
Maybe something like `numpy.typing.ArrayLike|numbers.Number`. Not incredibly precise. But if this covers also pandas.Series (not entirely sure), then I think it would cover 99.9% of my practical needs.
numpy arrays are effectively the standard library datatype for matrices in Python land.
Type annotations are just hints and aren't enforced at runtime. You can pass whatever you want into that function, and you can also create a union type with the Fraction if you want the annotations to match your application.
Type hints don't help performance either because the interpreter discards them. The hints are not enforced by CPython, so you now need a build tool instead of just running the script. The whole point of PEP 484 seems to be to enable autocomplete in Visual Studio Code.
> The whole point of PEP 484 seems to be to enable autocomplete in Visual Studio Code.
It's to enable tooling, but not just autocomplete. The main purpose of typing is preventing type-related bugs, though making editor autocompletion and other editor information more robust is a not-insignificant additional benefit.
PEP 484 (created 29-Sep-2014, accepted 22-May-2015) wasn't motivated by VS Code (announced April 29, 2015).
PyCharm understands type hints for years now. Without using mypy btw. VSCode should be able to that, there should be one or more (either finished or half baked) plugins for it.
Edit: Microsoft Python Language Server has it (i read)
It has nothing to do with any particular editor, it is for autocomplete and build time type checking in any editor and type checking in people's build chains.
Your code should get type checked whenever you run tests and can help reveal subtle bugs (for example, allowing Optional but not handling None in all code paths)
The word you are looking for is structural typing. (as opposed to nominal typing)
It's essentially type-safe duck typing. In the case of square, you say i want x and y to support the multiply-operator, but i don't care what type they are otherwise.
I was just thinking that squaring an imaginary number results in a real number - that's a pretty crazy thing for a type system to handle, no? I guess that means the correct annotations would be more along the lines of:
def square(number: numbers.Complex) -> numbers.Real: ...
T = typing.TypeVar('T', numbers.Real)
def square(number: T) -> T:
return number ** 2
Wow, this is pretty exciting. Hope we also see match being accepted in Python 3.10, so this would make (for me at least) one of the best Python releases in years.
>You can install the typing module all the way back to Python 2.7, where list[int] will never work but nothing prohibits typing.List[int].
Why though? Even in 2.7, isn't the : type and -> type part just taken as annotation (that is, could just be ignored and not clash because it's a valid type)? Or was it some parser shortcoming?
I'm confused, type annotations is not supported in Py 2 anyways, and if you're gonna use type comments a la `# type: list[int]` then it doesn't matter really, does it?
> One reason was so the type system could be backported. You can install the typing module all the way back to Python 2.7, where list[int] will never work but nothing prohibits typing.List[int].
The system has been rolled out slowly in general, without hasty changes to the core language.
If only they'd had that idea a bit earlier in the Python 3 life cycle...
No it isn't. In a static type system, you're not relying on annotations because the types are all known statically. But in a dynamic type system, you can have regions of untyped code where the best your third party static analyzer can do is slap "Any" on them and you watch as the precision of your type checking erodes to nothing.
I don't know if mypy allows you to mock the API but at least flow does for javascript. If a library is untyped you get a file with all the exported functions and you can quickly set the types for the functions that you interface with. We have been using this for a couple of years and it's a godsend. There's a community project to share these external library definitions https://github.com/flow-typed/flow-typed
An annotation is when you write some information and it doesn't directly impact the evaluation of that which it annotates. It's a "note", essentially. Type annotations in Python do not actually affect any aspect of runtime and are ignored during evaluation. This is why you need a third-party tool to look at them. You can actually stick any arbitrary code you like after that colon — it's all ignored just the same.
What you're referring to are called type ascriptions, and they are often seen in what are sometimes called manifest type systems, where you must explicitly write out your types within a static type system. C and Java are common examples of this. These are not annotations because they have real meaning (either used for declaration or for type conversion).
Additionally, there are plenty of statically typed languages which do not require ascription. These languages feature type inference, which is a process that allows the programmer to leave the deduction of the actual types to the compiler. Haskell, Swift, OCaml, Scala, and I think now C++ to some degree (with `auto`) allow for this (to name but a few).
> Type annotations in Python do not actually affect any aspect of runtime and are ignored during evaluation.
This is equally true in Java. Types are erased at runtime. They don't exist in the bytecode.
Just because the type checking and evaluation are done using one tool, "javac" doesn't really mean anything.
Again, what's the difference between a bash script that runs mypy prior to invoking python and a bash script that compiles and runs a java program, from the perspective of type safety, assuming you know nothing about the two programs?
There is no difference. Neither provides any more guarantees than the other. Put another way, by enforcing use of mypy prior to invocation, you convert the annotation to an ascription.
A deeper analysis would suggest that "ascriptions" are actually the things you use to disambiguate when type inference is not powerful enough (https://docs.scala-lang.org/style/types.html).
Scala, for example, calls its type declarations "annotations", except when they are needed to disambiguate, when they are "ascriptions". So under the scala definition, python's annotations are exactly the same as scalas as long as they are used for type checking, which, if you use mypy, they are.
> This is equally true in Java. Types are erased at runtime. They don't exist in the bytecode.
For the record this isn't true, as anyone who's seen a ClassCastException knows. A subset of type information is erased under limited circumstances, but a lot of type information is still present in the bytecode.
> Just because the type checking and evaluation are done using one tool, "javac" doesn't really mean anything.
What means something is that the types are part of the language specification and, even more importantly, part of the consensus of the language community.
In particular, if you depend on a Java library, you can be extremely confident that the type information of that Java library is actually correct, because you can be extremely confident that the library was built in a way that involved checking the types. This is simply not the case for Python; it's very normal to build Python without running mypy. It's like arguing that C is a safe language because you can run Valgrind on your C code: yes, that external tool exists, but it's not part of the language by default, and so you can't rely on the rest of the ecosystem to have used it.
> A deeper analysis would suggest that "ascriptions" are actually the things you use to disambiguate when type inference is not powerful enough (https://docs.scala-lang.org/style/types.html).
> Scala, for example, calls its type declarations "annotations", except when they are needed to disambiguate, when they are "ascriptions". So under the scala definition, python's annotations are exactly the same as scalas as long as they are used for type checking, which, if you use mypy, they are.
The distinction is a lot more general: type ascriptions change the behaviour of the code while type annotations do not. So a system that only has annotations and not ascriptions is significantly more limited than a full type system that has ascriptions as well, at least in practice - in theory we can do perfect type inference (e.g. H-M), but that ignores some edge cases that are important for real-world programming e.g. handling literals in the source code.
> part of the consensus of the language community.
Indeed, so the only difference is that you trust one languages annotations more than the others. Which is what I said.
> The distinction is a lot more general: type ascriptions change the behaviour of the code while type annotations do not.
Well no, ascriptions don't change behavior, they remove ambiguity when systems aren't able to infer things on their own. Literals aren't the issue for H-M. They're relatively easy. H-M has trouble with inheritance and heavily polymorphic code.
> Indeed, so the only difference is that you trust one languages annotations more than the others. Which is what I said.
Eh, that's like saying there's no difference between breaking the law and not breaking the law, someone might kidnap you and lock you up either way. It's true in a sense, but it's more misleading than helpful.
> Well no, ascriptions don't change behavior, they remove ambiguity when systems aren't able to infer things on their own.
You're trying to gloss over everything as "removing ambiguity", but by that logic any change to a program is just "removing ambiguity". In general there are other possible ascriptions for a given term that would form valid programs, and in general those programs could have arbitrarily different behaviour from the program in question.
> You're trying to gloss over everything as "removing ambiguity", but by that logic any change to a program is just "removing ambiguity". In general there are other possible ascriptions for a given term that would form valid programs, and in general those programs could have arbitrarily different behaviour from the program in question.
I'm not, for what it's worth. There are cases where you could argue that type annotations are just removing "ambiguity" (the type system is successfully inferring, you place an annotation that agrees with the inference, or a stricter annotation that still checks successfully). I can see how this is an ambiguity, but it isn't what I meant.
But scala's definition of ascriptions are for places where the type system can't figure something out, and needs extra information to successfully check. More powerful type inference might address these issues (most common languages don't use H-M). Ambiguity was perhaps the wrong word to use. "aren't able to infer things on their own" is maybe the more important bit.
Type ascriptions don't "change" behavior, because there wasn't valid behavior without them. The type system failed to check (but again, different inference algorithms might be able to get around this)
> Type ascriptions don't "change" behavior, because there wasn't valid behavior without them. The type system failed to check (but again, different inference algorithms might be able to get around this)
No, they exist to change behaviour, that's the whole point of that terminology. The example they give is passing a sequence to a vararg method: if you have a generic method that takes a vararg (generic) parameter, then calling that method with multiple parameters from a sequence, or a single parameter that is a sequence, are both valid but have different behaviours.
I totally agree with everything you've said in this thread. The annotation/ascription distinction comes down to whether the use of the thing changes anything in the result. (Just adding a voice of support in this discussion, for what little it's worth.)
> This is equally true in Java. Types are erased at runtime. They don't exist in the bytecode.
I work with bytecode on a daily basis (I develop a commercial Java obfuscator) and this is untrue. You may be thinking of generic type parameter erasure, but even then special interface methods are generated by the compiler so that no lowering to java/lang/Object happens too soon.
In non-generic code, the types remain and are strongly enforced. Methods have descriptors that restrict parameters to certain types (you can get a VerifyError upon loading the class if you've generated code with a type confusion), there is a 'checkcast' instruction to ensure that a stack values is a certain type, and a ClassCastException is thrown if a cast is impossible.
Alright, we've got two different discussions going on here. I'll address them separately.
---
As far as "annotation" vs "ascription", you've actually just reinforced what I said.
> This is equally true in Java. Types are erased at runtime. They don't exist in the bytecode.
Types are erased at compile time, which means that the type-checking analysis has completed. Many compilers erase types after doing type checking to improve runtime performance. That doesn't mean the types don't do anything. They're used for guiding the type-checking by the compiler.
This is in contrast to Python, which does literally nothing with the stuff you throw in the annotations. MyPy doesn't play a role in this specific aspect of the discussion because we're talking about language implementations, and in Python the type annotations are just artifacts in the code that get ignored during compilation and evaluation.
This is the distinction between ascriptions and annotations, which you actually corroborated here:
> Scala, for example, calls its type declarations "annotations", except when they are needed to disambiguate, when they are "ascriptions".
All of the type stuff in Python is handled via annotations because the language proper does not make use of them whatsoever. They're just decoration, really.
In contrast, Java uses all the types to perform type-checking, making them ascriptions.
In languages with lots of type inference, like Scala, you can supply type information. Sometimes these are ascriptions (which either help the inference algorithm or are used for casting), and sometimes they are just artifacts to help the programmer (in which case they can be regarded as annotations, which I didn't address previously).
But the key component in all of this is that we are only looking at the languages proper, not the suite of third-party tools available to those languages, because this is what dictates the correct terminology. In Python the language, types are just annotations which are completely ignored, and this is not the case in statically typed languages.
---
Which brings us to the second point: your original argument.
This thread started with somebody asking whether static typing will ever be implemented in Python proper, to which somebody said there's no reason for that when we can use MyPy and disable support for `Any` types. The response to this was that it would require all third-party libraries to be fully and correctly annotated, where you came in and claimed "that's true in any type system."
The reason I disagreed with this is because in statically typed languages you're not relying on annotations for anything. Keep in mind that we've established that annotations are specifically artifacts in the code that the language itself doesn't care about, because when the language does care they are called ascriptions. Annotations are supplied by the programmer by desire, not by necessity of the language proper.
You can release a fully-functioning Python library with absolutely zero type annotations, and it will run exactly the same as if you annotated everything. This is because type annotations don't do anything in Python; to gain any use from them requires using third-party tools, like MyPy.
But you cannot release a Java library that doesn't have all its types specified in some manner. Because the Java language requires the types to be present, meaning they aren't annotations like they are in Python.
I guess really what I'm trying to get at is that your response where you said "That's true of any type system" was completely missing the point. The point being made by the person you responded to was about how in a Python ecosystem, simply using MyPy and disabling the `Any` type requires all third-party libraries to be fully and correctly annotated — something which is not usually required in Python, so it's an extra burden on the libraries' developers. This is entirely distinct behavior from any statically typed language because in those languages, you can't just leave out the types and call it a day.
> In contrast, Java uses all the types to perform type-checking, making them ascriptions.
Javac uses them to perform type checking, prior to runtime. Exactly the same way that mypy uses them to perform type checking prior to runtime.
> The reason I disagreed with this is because in statically typed languages you're not relying on annotations for anything. Keep in mind that we've established that annotations are specifically artifacts in the code that the language itself doesn't care about, because when the language does care they are called ascriptions. Annotations are supplied by the programmer by desire, not by necessity of the language proper.
But Scala's annotations are used for type checking. Much as python's are if you enable mypy.
> But you cannot release a Java library that doesn't have all its types specified in some manner. Because the Java language requires the types to be present, meaning they aren't annotations like they are in Python.
Of course you can. You can write a functional java library that specifies that all functions take `Object`. Your argument comes down to that syntactically, java requires you stick something in the place-where-type-declarations-go, while python does not.
This ultimately doesn't have any impact on the strictness of the type checking done, because you can stick useless annotations (Any, Object) in the syntactic spot in either case. The semantics are the same.
Ultimately, your argument comes down to "I trust that more third party libraries will have useful types in Java than in python", but that isn't implicit to the type system, it's because the type ecosystem has been around longer.
Which, like, sure. But to say that python doesn't have static typing is silly. Nearly all the python I write is statically typed.
> In Python the language, types are just annotations which are completely ignored, and this is not the case in statically typed languages.
This is a uselessly semantic argument. Much as if you choose to not typecheck your python, you can choose to write essentially untyped java using Object and casts. Are you really certain that no library you depend on does that, or uses reflection to access dynamic and non-statically-verifiable features?
The safety you get from any static analysis tool, whether a type system or a linter or whatever, relies on your dependencies not trying to subvert the tool (or doing so correctly).
This is essentially the argument about rust's Unsafe. You, or any of your dependencies, is free to subvert the borrow checker and do "unsafe" things.
You either have to trust that they are doing so responsibly, or manually verify the correctness of their unsafe code. Having to opt-in to unsafe access is, indeed, an advantage since it makes static analysis of how much "unsafety" there is easy (much as it's easier to find non-static things in vanilla Java than in python), but you can (and some do!) use static analysis to enforce type annotations in python throughout the entire code base.
> so it's an extra burden on the libraries' developers.
I'm sorry, but it is your argument that is uselessly semantic.
Yes, people could write Java code where all the type ascriptions are for Object. It is technically possible. But nobody does this. It has nothing to do with "the type ecosystem has been around longer" — it has to do with the fact that the types do something in Java, and always have and always will. You cannot compile a Java program and just not check the types.
But people do write Python libraries without type annotations, and that's fine. There's plenty of not-explicitly-typed Python in the world because that's all there was for a long time before the introduction of optional type annotations. And a lot of people prefer to write Python without type annotations at all because that's one of the benefits of a dynamically typed language — explicit types are unnecessary.
I think your argument is useless because you're intentionally conflating these things and arguing that people could write type-less Java, but since this is not actually done in practice it has no bearing in this conversation.
The point originally being addressed was that in Python you either need fully annotated libraries or else you cannot simply "use MyPy with Any turned off" as the far-parent comment suggested. And my point has been that this is a genuine concern in Python because there's lots of Python code without annotations because Python the language doesn't require or even use the annotations, so they are (from the perspective of the language) purely artifacts of no consequence. This is not the case in real, actually used Java code.
Please stop making arguments in bad faith to "prove a point" or whatever. The point the entire time has been about the use of type annotations in practice, and you appear to be intentionally disregarding this in an attempt to... I don't even know. What are you after here? Do you just not like that somebody disagreed with you? Like, I'm genuinely unsure what your purpose is other than to be contrarian. The concern about "are all the third-party libraries I'm using fully annotated?" is a genuine one in Python that is never even thought about when writing Java (have you ever paused to wonder this when writing Java?), so your argument amounts to a semantic debate that has no basis in the real world.
> This is not the case in real, actually used Java code.
But is the case in its type system, which is what I said. If you want to argue that the python developers are less prone to typing things than the java devs, then say that. Don't claim wrong things about the type systems. They're functionally equivalent.
The type checkers have all the same information, however much the users choose to give them. There are actually a few ways in which python's type system is fundamentally more capable than java's (literal types, protocols).
Literally all a static type system is is an assertion (and tooling to assert) that types must validate before you can run the code. That's it. If you provide a presubmit check that types must validate before you run the code, congratulations, your code is statically typed.
Put simply: enabling mypy + no-any gives you all the same static guarantees (and arguably more) that java does, so in what way isn't it statically typed?
> Put simply: enabling mypy + no-any gives you all the same static guarantees (and arguably more) that java does, so in what way isn't it statically typed?
The original question was about Python the language, not Python the ecosystem. At no point is it ever likely to be correct to say "Python is statically typed"; it will only be possible to say "this specific Python code can be statically typed, if you run the static type checker". This is a fundamental difference between optional static typing in Python versus a natively statically typed language. Because you can run the Python code without running the static type checks, which you cannot do in, say, Java.
Because of this distinction, it is never a genuine concern of a developer in Java or another statically typed language whether third-party libraries will be properly typed. There do not exist libraries without valid static types. You can't install something from Maven or what-have-you that won't type-check.
But you can do this in Python. You can install something via pip and there is no guarantee it will come with good type annotations for your type-checking tool to use, because the type checker is not part of the language proper. It's not required. So Python code exists in the real world that cannot be used with the same assumptions as you can use code in the Java ecosystem. And this is fine, by and large, because Python wasn't meant to be typed. There's nothing wrong with using code like this, nor is there anything wrong with writing code like this. You may not wish to do so personally, which is fine, but I don't think there will ever be an expectation made by the Python language proper of all Python developers to annotate their code. Python will always work without explicit types.
My point in all of this is that your original argument that "[whether there are concerns about third-party libraries having valid type annotations] is true in any type system" does not hold in the real world, because you simply won't have untyped libraries in a statically typed language's ecosystem like you will in Python. Perhaps you didn't realize that this was the point being made, but in any case you went off on a very semantic tangent that had no real bearing on the conversation at hand: the use of type annotations in practice.
> The original question was about Python the language, not Python the ecosystem.
Python doesn't have a spec, so the argument that the language doesn't support static typing is dubious at best, since a static type checker is maintained by the language itself. If I alias python to `mypy $0 && python $0` or whatnot, is it no longer python? It's certainly statically typed.
> Because of this distinction, it is never a genuine concern of a developer in Java or another statically typed language whether third-party libraries will be properly typed. There do not exist libraries without valid static types. You can't install something from Maven or what-have-you that won't type-check.
Is this enforced by the language, or is this just an aspect of the ecosystem? "Maven-packages must typecheck" is enforced by the spec?
> But you can do this in Python. You can install something via pip and there is no guarantee it will come with good type annotations for your type-checking tool to use, because the type checker is not part of the language proper. It's not required.
Are you saying that there is a guarantee in Java that the types need to be "good"? I'm not even clear that maven (which again isn't part of the java language) requires that uploads compile, much less that the type declarations be "good", whatever that means.
Which all returns to my central point: you trust the java ecosystem more that the python ecosystem to have "good" type declarations. There is nothing implicit in the type systems of those languages that enforces that one or the other have better or worse type annotations.
Your argument relies on conflating use in practice with theoretical guarantees, I'm just pointing that out. If you want to say that your experience is that python has less reliable types than java, then sure, that's probably true. But the type systems available in those languages are functionally equivalent. The use, however, differs.
It really feels like your entire contribution to this discussion amounts to "but what if we ignore the point being made and instead just focus on very strict semantic interpretations of the language other people in the conversation are using". I feel like you have not really engaged with my actual points and instead have focused on finding details to complain about in service of appearing to make a point, which effectively derails the true conversation. I find this incredibly frustrating, and I have no interest in continuing.
I would politely suggest you consider whether this is a common issue you have with people or whether it's just me (certainly a possibility, I admit). If this happens to you often, you may want to try alternative approaches to broaching your points if you'd like better engagement. I think most people don't enjoy when their main points are continually ignored in favor of bringing up other minutiae to prove that they're wrong or something.
To clarify, no, this is a complete non-sequitor. The ability for a type system to do inference has no bearing on whether or not the types are checked or not. To make things even more confusing, as an example, python can have type inference (pytype) or not (mypy). In either case, annotations are type checked.
The other discussions have already pointed out some limitations in your interpretation (heh) of types.
I think the key difference between the annotations in Python and types in Java, or even more strongly in languages like Rust and Haskell, is how they are used in the language ecosystem. In Java and Rust, the types are cental to how programmers think in those languages. A large part of the dev cycle goes into ensuring the types are consistent. In python, these type annotations are a relatively recent addition that the community has not completely adopted yet. A lot of real-world python code will have no types, thus the (3rd party) python type checkers will fail to ensure the benefits of having strong consistent types.
And how can we then justify "types" like Object in Java or Any in Rust? Well, sometimes we do want to write code where we want to tell the compiler to stop checking types. However, these are more of an escape hatch rather than a central part of the language ecosystem. As someone else already pointed out, the Java or Rust community does not go about writing code with only Object or Any, even though they can in principle. But in python, that's what the community has been doing till now.
It goes the other way too. If the python community starts taking the type annotations seriously, it will start approaching the same usefulness as in these other static languages with good type systems. Then we can start expecting that the 3rd party library type annotations being accurate, which started this discussion in the first place.
Yeah sorry, I stopped reading after I felt the discussion was going round in circles. But it's funny how the same viewpoint results in different takeaways. Of course, the topic under discussion is a pedantic one.
However, "that's true of any type system" just feels dismissive of the current state of the world. The comment you were replying to was pointing out that at present, enabling strict type checking in Python, that has been bolted on as an add-on only recently, will not be able to provide benefits to the same level as the ones in Java and Rust where types are front and center. That might become true tomorrow when all Python libraries have adopted types. Hypothetically, tomorrow the Rust community could also decide to adopt Any as the central type everyone uses. But today, the state of the world is very different. Python is still known as a dynamically typed language where devs can essentially forget explicit types, and Rust is a strongly statically typed language where devs will rarely have to worry about missing types.
A technical note: Python is (relatively) strongly typed, but it is not statically typed.
And the lack of static typing is a fundamental feature of the language. I doubt very much whether a static type system will ever be implemented in Python proper.
It would probably never happen and I dont see why it should ever matter. You can always use tooling to enforce it as another comment mentioned. I would argue if you decide to type annotate a whole project you should just go for it.
I have been type annotating my project over time in the hopes I eventually have just enough context about what I am doing. Also a good IDE will show you any place wrong types are criss crossed.
The PEP abstract says this: "Static typing as defined by PEPs 484, 526, 544, 560, and 563 was built incrementally on top of the existing Python runtime and constrained by existing syntax and runtime behavior"
One of the reasons was that `list[int]` could not be done, as that would refer to a list index. Same as you would do
`my_var = [1,2,3]`
`my_var[0]`
Therefore it was possible to use just `int` as a type.
From python 3.9 (unless you used from __future__ import annotations in 3.7+) type hints are not parsed "as python", at function definition time, but more or less as text.
In 3.9 `list[int]` does not want an item from object `list` at index 'int' when using it as a type annotation.
The typing library did not have these issues because the `types` altered the way the objects worked. Probably (didnt check it!) by overriding __getitem__ and the likes.
Well, `list` and friends were functions to construct those objects. So presumably they did it this way to prioritize adding typing over getting it perfect.
It was a problem only because somehow they decided that type annotations should evaluate to something meaningful at runtime, instead of being a compile-time only construct like in every other language.
Wouldn't compiler selective enforcement achieve the same goal? It seems to me the difference is that other programs can be implemented to do the typechecking (eg mypy) rather than the compiler doing so itself
The runtime mirror doesn't have to have the same name as the compile-time type. Look at Java where you have to do weird things with class objects to do reflective access.
I suspect a big part is the general desire to minimize the runtime impact of typing support,especially on code not using it; there's a bunch of infrastructure added to the core classes that now can be generic types to support that.
Yes! The Olson Database is the Right Thing™. Timezones are a weird human artefact, but ordinary humans have no idea what ZQMT or whatever actually "is" and will make loads of egregious mistakes if exposed to these zones directly even if your UX isn't terrible (which it probably will be).
But humans do understand what a city is, and if they live in or near one they understand what the time is in that city, adjusting for any weird local political rules like "Daylight saving" because those rules affect their life.
It's the right granularity practically too. Can politicians decide that Dublin and Belfast are in different timezones? Yeah, it'd be difficult but I think they could make that happen. How about Queens and Brooklyn? Ahahaha No. The residents will defy any pretence that clocks change between one city block and the next, it would never stick.
Curious thing. The Chattahoochee River divides the US states of Alabama and Georgia, the former being on Central time and the latter being on Eastern time. There's a city in Georgia called Columbus, and it's right up against the river, and so is adjacent to Alabama and Central time.
Years ago we visited my wife's family in Columbus with some frequency, once we visited the Wal-Mart supercenter in Phenix City (yes, it's spelled way wrong) Alabama, which is right on the eastern border of Alabama, also next to the river. Since I worked in Wal-Mart Information Systems at the time, I had quite a bit of access and noted that the store's systems were on Eastern time, even though the entire state of Alabama is on Central.
Sure enough: https://en.wikipedia.org/wiki/Phenix_City,_Alabama "Phenix City lies immediately west across the Chattahoochee River from the much larger Columbus, Georgia and observes Eastern Time on a de facto basis (in contrast to the rest of Alabama, which observes Central Time) due to Phenix City's strong economic ties to Columbus."
The Olson Database is amazing; I've non-trivially interacted with it a number of times in my career.
Even so, the final, on the ground authority is the people who use and are affected by time zones.
Yes. When applications ask me what timezone I am in and simply give me the offset I get very nervous. Especially since they are usually event scheduling or email sending services where timing actually matters a lot.
This is an inner EU border, so similar to Phenix City[1] mentioned above.
Many people live and work on different sides of the border, but they just accept things as-is.
There is a problem with the timezone database, which is that it can be a little too granular, since it defines a timezone as all places that share the same clocks since the Unix epoch. This means that if jurisdictions decide to move from one timezone to another, they each get their new timezones.
Take Indiana. Indiana is in the US Eastern time zone, except for some places near the Illinois border, which are US Central. But Indiana also didn't observe DST until 2006 (although some counties did observe DST). Consequently, what could generally be described as three timezones (as in the MS timezone selector) is actually described as 11 timezones: https://en.wikipedia.org/wiki/Time_in_Indiana#tz_database
From what I understand, locals in Indiana actually refer to time simply as Eastern or Central (with non-observance of DST meaning you shift from Eastern in winter to Central in summer).
Cool. Will this obsolete pytz I wonder? It does seem like quite an important thing to be in the standard library given the kind of things Python is used for. It's amazing how few developers actually understand timezones.
I found the Pendulum library [0] very intuitive when it comes to working with time. It makes it easy to do the right thing. For example, if you call `pendulum.now` it gives you a timestamp including your local timezone information. In other words, it preserves "the time on the clock on the wall", which is lost if you do what so many people do and just take UTC timestamps.
– “Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive.”
– “Python now gets the absolute path of the script filename specified on the command line (ex: python3 script.py): the __file__ attribute of the __main__ module became an absolute path, rather than a relative path.”
On the last one I rarely if ever use __file__ as a relative path and have definitely typed os.path.abspath(__file__) many many times.
> – “Python now gets the absolute path of the script filename specified on the command line (ex: python3 script.py): the __file__ attribute of the __main__ module became an absolute path, rather than a relative path.”
Good lord, finally! The number of times I've tripped over this.
And when it’s coming via a symlink. And on a networked file system that’s been remounted a couple of times. And parts of it are bound to other parts of it. FML.
> Note: On file systems which support symlinks, the directory containing the input script is calculated after the symlink is followed. In other words the directory containing the symlink is not added to the module search path.
nice, removeprefix/suffix is so pythonic. Great addition I've seen handrolled implementations of that go buggy because they happened to work in some cases but broke others.
"There have been repeated issues on Python-Ideas [2] [3], Python-Dev [4] [5] [6] [7], the Bug Tracker, and StackOverflow [8], related to user confusion about the existing str.lstrip and str.rstrip methods. These users are typically expecting the behavior of removeprefix and removesuffix, but they are surprised that the parameter for lstrip is interpreted as a set of characters, not a substring."
No, this is exactly the common coding mistake removeprefix/suffix is here to adress. The strip() family removes all characters in the string you pass, so it removes not just "prefix" but also "efix", "prefixix" and "effxpixpefiii".
meta, but I upvoted this comment because it illustrates the exact reason why these functions were added. It adds to the conversation in an important way. Is that not the reason for upvoting in general?
That's kind of complicated, I think people generally use upvotes both to signal that something is wrong, and to signal that the comment detracts from the debate.
This case is actually an interesting edge case - I think downvoters are of the mindset that the comment might spread disinformation, therefore it's better to hide it. But I actually agree with you here - this is something a lot of people think, it is wrong, and it's better to explicitly have people see the wrong information, and learn why it's wrong. That's why the PEP itself mentions this as a common misunderstanding!
They do mention that it’s more efficient than lru_cache, which isn’t surprising since cache is a lot simpler (just throw all the values in a dict keyed by the args).
I find it weird that they called it `cache` instead of `cached` which would be more in line with `cached_property`. One of my colleagues already tripped over this today when he told me about this new feature.
I have mixed feelings about a 12 month development cycle. From the perspective of a library maintainer but not a core contributor, this seems like a headache to support up to five versions concurrently. I understand we want to avoid the Python 2.7 debacle, but seems pretty rapid no?
The performance table seems to show a huge amount of regression -- a notable example, reading bound methods is almost twice as slow vs 3.8. Does anybody know if this is normal for early releases like this; will we see those numbers improve over time?
It must feel great to be a Python Core Developer. A 1% perf improvement in any part of the language translates to billions of CPU hours saved around the world.
As PHP discovered, JIT is not a magic button. If you don't have efficient data structures and memory use, JIT won't do much for code that isn't just raw number-crunching.
Do we have full subinterpreters support yet? It appears this update impliments seperate schedulers now, but not sure if that means we can run as independent threads or if they're still bound by the main thread.
As a sysadmin who needs to prepare literal thousands of finicky packages with each major version, I'd love to return to the C release cycle of ~10 years.
I'm with you there. A programming language is my tool, not my destination. I shouldn't have to keep learning how to hold my hammer every six months nor have to go back to every house I ever built to replace the nails every year.
There is value to being able to say "I'm done" and walking away knowing my code will happily chug along for the next 20 years.
Yeah, interesting reading through the list of "Removed" features. On one hand, it is nice to tidy things up, on the other hand, I'm wondering, was it really worth breaking backwards compatibility for these things?
Ha... I had asked a question in Quora several years back on exactly the same topic (deleted account, don't have link).
I don't understand why programming languages need to be released like other "software". I see it a lot for modern languages like Rust.
To me, a programming language should have a major revision only once in 5-10 years. All other changes should be performance improvements or implementation changes which do not affect the end user which in this case is the programmer.
I would like to know the details, seriously. Thousands of broken packages because of major python release?
As a former package maintainer I can assure you that every update of gcc breaks the build of some other package. And we are talking about C or c++, languages existing longer than Python and having much stronger backward compatibility story. And no one ever said that upgrade to recent gcc is a problem.
Iow there is hardly any up to date system which do not require constant maintenance.
You should just let developers use virtualenv and then let them install packages through pip, or let them use tool like poetry which makes the experience even better. This makes life easier for both sides.
If you use RedHat or CentOS I would highly recommend ius.us repo, it contains latest versions of python and it's pretty much all what's needed.
Windows 7 still has 25% market share, apparently, at least according to https://netmarketshare.com/operating-system-market-share.asp... (no clue if that's a reasonable source or not, but probably not TOO ring). Seems a little premature to drop support?
No, Windows 7 is still supported until 2023. At at minimum for all companies that are paying for windows support automatically.
As a consumer, I have an old Windows 7 laptop that I use once a month or less (don't need to travel much lately). I can tell you that it's been continuing to run windows updates all year long after the EOL. It's a major issue for me to use the laptop because it's running updates for hours whenever it's taken out of storage :(
Sucks for some individuals, but I guess those companies paying for support for Windows 7 can do the same for python support if they require it. At some point those volunteering their support probably want move on.
I can't say how much of a nice improve removeprefix() and removesuffix() is. lstrip and rstrip prefix and suffix trimming bugs crop up in people's code constantly. This seems like a nice change, especially since the alternatives are so ugly.
They got rid of the super ugly [:] but introduced a new ugliness.. | and |= operators for dictionaries. Why is this so important that this needs special operators? Python should be readable by everyone who can understand English. I hate when languages keep inventing hieroglyphs, twisting my fingers to type |= takes even longer than typing .join.
as pointed out elsewhere in this thread, `"a" or "b"` evaluates to "a", which kind of makes me think that `a | b` (with a and b as dicts) would take values for overlapping keys from a, not from b.
The dict literal with dict-splats conveys a sense of ordering that makes the behavior of taking values for overlapping keys from the lattermost dict make more sense to me.
No? I think the asterisks (two of which acts as a “pointer” to an expanded dict, instead of the dict itself) is straightforward and yes more aesthetic for my taste. Not sure what the circle jerk of non-sequitor comments and downvotes is for?
I find this operator intuitive and idiomatic. It's widely understood and used as a binary OR operator in many languages, and it's easy to understand its functionality when both operands are dictionaries.
It’s also already an established part of the set syntax - it’s been possible to ^, | and & sets for a while now (symmetric difference, union and intersection). While not all of those make sense for dicts, | for union does and is quite intuitive.
It's also consistent with the set notation, which makes it easier to remember. Dict keys already kind of work like a set, this extends the behavior to deal with the values as well.
This is my favorite change in this release and cause me to upgrade to the RC version a few days ago. It's a huge QoL improvement for so much code I write.
I'm also a fan of the walrus operator. These are small, rare, clear, useful, idiomatic new features, in my opinion.
I personally feel walrus operator is a much more prominent improvement, considering how prevalent it can be used (like, I use it in almost every single `re.search()`, and at least half of conditional list comprehensions and `while`), while I don't find the old way of updating dict (`{a, b}` etc.) too bad.
I feel the same way about -> in JavaScript. You would think after all these years I'd have come around by now. Nope, I'm still pissed off about that decision.
The parent comment referred to bugs from use of lstrip and rstrip for this purpose. The problem is that they often appear to work for this job, so they can make it all the way into code. Your own example strings illustrate that:
> "foobar".lstrip("foo")
"bar"
Of course, if the developer making this mistake tested with "foofbar" instead then they would realise their mistake. A more typical example of this type of mistake would be url.lstrip("http://").
0 == False, right? So any set-like container that uses == and not `is` for equality checks would have to only ever include one of 0 and False. This seems fine to me.
Yeah I wish they didn't let you do arithmetic on Boolean with int without explicit conversion. It can be useful, but the explicit conversion isn't a large burden and prevents many errors.
Lots of great improvements. Sadly I won't get to enjoy the new string methods, dict operations and generic hinting as I'm still stuck writing code that should be compatible with older versions of 3 (and 2.7 but six takes care of that).
It operates similarly to bitwise or, combining all the set bits, or like the or operator on sets, combining all the set elements. "|" has never meant logical or in python nor in most other common programming languages.
[Edit: I thought the parent comment was saying that they were expecting dict1 | dict2 to equal either dict1 or dict2, not some combination of them. Instead, they were referring to the which value gets chosen if the same key is in both dictionaries. Thanks to minitech for their comment that helped me see what they were saying.]
Are you also surprised when the original bitwise meaning of "|" does not return either of the two arguments?
Bitwise or works at the level of bits, so the question is more like, in:
>>> bin(0b011 | 0b010)
'0b11'
Does the leading 1 come from the first argument or the second argument? It isn't really a sensible question for bits, since the 1s are indistinguishable. However the values in the hash table are distinguishable, so it's a valid question.
Also, taking an example from another comment:
>>> {0} | {False}
{0}
>>> {False} | {0}
{False}
This is another example where the union operator is used, but here it picks the first value, rather than the second value.
Gotta say though that I intuitivly think of the dict union operator x | y as {* * x, * * y}. Which is how I would have joined two dictionaries before, when I wanted to produce a complete copy of one, i.e. not use dict.update.
According to the Guideline "please use the original title, unless it is misleading or linkbait; don't editorialize" the Title should be "What’s New In Python 3.9"+
Right now, the link title is not even that, but just "Python 3.9". I wouldn't be surprised if it was an after the fact change; I've observed that HN admins never let "X Y.Z Released" or "What's New In X Y.Z" headlines stand but change them to "X Y.Z" or even just "X" almost immediately.
It's a strange HN-ism that puzzles me - "OK, what's so important about Foo 3.14?" There is a rule that is (barely IMO) on point though in the posting guidelines you linked to:
> If the title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
sums = [s for s in [0] for x in data for s in [s + x]]
Why would you do "for s in" twice? Is that intentional? It would make more sense to me if the variables would have been different. And why would you want to add 0 to numbers?! Curious about a real world use case for this.
I wouldn't endorse that code, but it does make sense. You can read it like this:
sums = []
s = 0
for x in data:
s = s + x
sums.append(s)
`for s in [0]` assigns 0 to `s`, as an initial value. `for s in [s + x]` adds `x` to `s`. Both instances of `s` are the same variable, there's no shadowing going on.
Ah, I see. That really doesn't deserve to be called an idiom, it's a clever hack. But it's nice to know about it. It seems less ugly than the walrus operator to me, and it doesn't leak the variable outside of the comprehension.
Flexible function and variable annotations. Oh, I feel we're going to have so much fun together.
That's probably gonna be very useful when reusing types with different libraries (such as ORMs or serializers). I think anyway, gonna have to experiment.
a number of Python builtins (range, tuple, set, frozenset, list, dict) are now sped up using PEP 590 vectorcall;
I wonder how much vectorcall improve performance in standard code
We’ve not had a chance to properly benchmark because one core dep isn’t compatible but the benchmarks I’ve seen suggest that it’s actually slower overall, but that’s expected to improve.
I wonder if topological sort being built into Python will increase the prevalence of more obscure graph problems for companies that make search engines and the like.
> Using NotImplemented in a boolean context has been deprecated
I've always wonder why NotImplemented is used in a boolean context in the first place, iirc when it evaluates to True the overrided comparison now becomes a source of hard to spot bug
Python's LL(1) parser isn't particularly fast. It's not a recursive-descent parser custom-made for Python's grammar, but a data-driven general-purpose design. What's more, it's not driven by a simple LL(1) table but by more complex data: a set of DFAs.
print "Hey, they brought back the print statement!"
print("But you can still call the print() function!")
print ("printing", "tuples", "works", "like", "python2")
It would be a massive QoL improvement and also accelerate Python 3 adoption.
(I'd actually be OK with allowing keywords to be used as method/function names in general, but that's a more extensive change than
I'm so extremely glad they got rid of the print statement. Was an ugly part of the language, and added keywords to python that didn't need to exist. I think they should get rid of the assert statement too and turn it into a builtin.
The print statement is/was a beautiful thing that dates to antiquity (e.g. Python 1 and BASIC), and one that many people loved. It was also used extensively.
Removing it (and breaking compatibility in general with no easy workarounds) is one of the major footguns in Python 3 that drastically slowed its adoption and basically forked the language.
It's a prime example of ideology trumping usability and practicality, resulting in the waste of probably millions of programmer hours, and orphaning millions of lines of code in Python 2 programs and libraries.
Other languages have managed to evolve effectively for decades without breaking backward compatibility; it's disappointing and embarrassing that Python simply hasn't.
History/preference aside, I think you’re being a bit hyperbolic regarding the cost of removing the print statement. It’s like the most trivial broken feature to fix with 2to3.
I’ve definitely run into issues 2to3 couldn’t automatically fix (encoding issues in some py2-only lib I needed to use IIRC), but upgrading print doesn’t really have much overhead.
2) This wasn't a problem until >> was added, because of problem #2 - the print statement didn't originally let you print to anything other than stdout. Which meant that if you wanted to support writing to a file instead of a stdout then you had to re-implement print yourself.
In practice, I've also found that having print as a function adds functionality because I can do things like:
print(*data_values, sep="\n")
which is a quick way to print each value in a list on its own line.
"Readable" is in the eye of the beholder, which is why I commented that this was something I personally found useful, rather than one of the two subtle flaws I mentioned.
I can also disable all print statements with "def print(* args, * * kwargs): pass" at the top of the module.
I've had times where I couldn't figure out where a print was coming from, so I could replace print() to check the arguments passed in:
import builtins
builtin_print = builtins.print
def my_print(*args, **kwargs):
if "looking for" in args: # adjust as appropriate
1/0
return builtin_print(*args, **kwargs)
builtins.print = my_print
Previously I had to do that by wrapping sys.stdout with my own file-like object, to intercept write(). (Granted, not hard, but harder.)
The fact that you can overwrite built-in functions is not a point in favor of the language. Imagine you do this in a library and everyone using print() in their project would use a modified print instead. You could've just modified sys.stdout instead.
This. It's hilarious how the parser knows what you're trying to do when using print statement in ipython and such, but still raises an Exception instead of printing the damn thing. Am I in the REPL to work on something, or to be frustrated by the ridiculous syntax? Just DWIM, Jesus Christ.
Now you can use list, dict, tuple instead of typing.List, typing.Dict, typing.Tuple.