Mostly, it's Java written with Python syntax, specially the gang-of-four patterns. I recall it being even more java-like, so, progress?
The stuff that is actually idiomatic python is either too convoluted or rather obvious.
I know this comes across rather mean, but this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples). There's maybe 2 or 3 good ideas in there, but even those I might be hard pressed to find a real usecase for.
I wholefully agree, the problem with this kind of repos is that Python already makes most patterns trivial. So you either create a trivial example that does not bring value or you introduce unnecessary abstractions for the sake of them.
I am always looking into these repos though because I am still interested in learning new Python tricks.
One of the repos that I found interesting recently is https://github.com/dry-python/returns
All these lambdas seem excessive. "Maybe"-like pattern in Python can look like (None is falsy and custom objects are true):
# note: `None and f()` returns None without calling f
balance = user and user.get_balance()
credit = balance and balance.credit_amount()
discount_program = choose_discount(credit) if credit and credit > 0 else None
Nullable (Optional[Foo] / Union[Foo, None]) and Maybe[Foo] are similar but not quite the same thing. The difference is quite subtle.
Optional[Foo] is same as Foo | None, meaning you can operate on foo with Foo's methods, except that if it's None, you get NoneType errors.
Maybe[Foo] is an actual container. You have to map over or unwrap it to operate on Foo.
The big difference is ergonomics/opinion. You can't actually map over Optional, so every time you wanna use it, you have to manually check if foo is not None. Whereas Maybe, you can operate generically over it. Some folks think "is not None" is better. Personally I hate NoneType errors in prod and find that much more painful than a bit of indirection.
It looks like a distinction without a difference. We can consider `None and f()` pattern as explicit syntax that unwraps objects as necessary without infecting other code.
Background: an object that either None or true in a boolean context (true unless overriden for custom objects).
Given such object, we can consider it in a virtual Maybe container/box.
When we want to use it, we have to unwrap it using `obj and obj.method()` syntax.
Then `obj.method()` is ordinary "unwrapped" call.
Just to remind you. Here's how "ergonomic" Maybe variant from the article look like:
# Type hint here is optional, it only helps the reader here:
discount_program: Maybe['DiscountProgram'] = Maybe.from_optional(
user,
).bind_optional( # This won't be called if `user is None`
lambda real_user: real_user.get_balance(),
).bind_optional( # This won't be called if `real_user.get_balance()` is None
lambda balance: balance.credit_amount(),
).bind_optional( # And so on!
lambda credit: choose_discount(credit) if credit > 0 else None,
)
The dry-python lambda soup is awful, I totally agree there. That's just there mostly for demonstration purposes though. Generally you'd actually use the flow construct (monads) to compose methods.
But `obj and obj.method()` really is not the same thing as `obj.map(method)`. "virtual Maybe container/box" is a nice idea, and does actually type-check with mypy (mostly), but you cannot actually compose it with other functions. The problem is, each time you do `obj and obj.method()`, you end up union-ing type(obj) and type(obj.method).
main.py:40: note: Revealed type is "Union[__main__.User, None]"
main.py:41: note: Revealed type is "Union[__main__.User, None, builtins.float]"
main.py:42: note: Revealed type is "Union[__main__.User, None, builtins.float]"
main.py:43: note: Revealed type is "Union[__main__.User, None, builtins.float, builtins.str]"
True Maybe types are more precise. Maybe if the Mypy engine could be retooled to recognise the `obj and obj.method()` idiom as tantamount to `obj.map(method)`, this could be avoided.
> I recall it being even more java-like, so, progress?
I'd consider Java-like being a regression, not progress.
In my experience, most of the time when people complain about medium-large Python projects being difficult to reason about, it's because they wrote Java code in Python.
In my experience, most Python projects are quite hard to scale or maintain precisely because they weren't written with a proper domain-specific approach. You'll be pressed to find a medium size python project that uses a database and doesn't use the leaky SqlAlchemy/Django ORM abstractions where data and operations on data are "the same thing"; On the same vein, any medium sized project with dozens of, hum, "models" will lead to files with thousands of lines. And don't even get me started on using an HTTP application instance as a container for configuration, usually included in every other piece of logic under the sun.
You can easily guess I'm not a huge python fan (nor java, but it so happens I work daily in medium-sized projects in python), and I do have problems with the "idiomatic/pythonic way", but lets face it - python, while very expressive, is often so convoluted that static analysis of code isn't even possible. If your IDE is not able to quickly determine a given var type, you expect the average developer to be able to do it?
Python is - first and foremost - a prototyping language, and an excellent choice for glue code. For big applications, you will actually learn to enjoy the bitterness and complexity of Java over python, and small stuff like strict typing.
>t this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples)
As someone with experience (just not in Python) I don't mind if the examples are basic, as long as they're idiomatic. I'd prefer a list of very common tasks/problems solved in a pythony way vs some examples of unique and clever uses of python.
Is there a difference between a "clever one-liner that doesnt replicate well" and idiomatic in python? Just because something is idiomatic it doesnt mean its a good approach, nor a good implementation.
> Is there a difference between a "clever one-liner that doesnt replicate well" and idiomatic in python?
100% yes.
Idiomatic would be (to me):
1. things which python does naturally - for instance list expressions rather than manual creation:
managers = [
person for person in get_all_staff()
if person.has_direct_reports()
]
kind of thing, rather than:
managers = []
for person in get_all_staff():
if person.has_direct_reports():
managers.append(person)
So that's an 'idiom'. And a common one.
2. Things that Python makes natural / or less so - by the way it works.
For instance, because of the important whitespace, and no end brackets or `end` keywords or whatever at the end of `if` or function definitions, large nesting inside functions becomes very klunky to follow the logic. So idiomatic would be to use small named functions, and split things into smaller pieces once they become too nested.
> >>> import this
> The Zen of Python, by Tim Peters
>
> Beautiful is better than ugly.
> Explicit is better than implicit.
> Simple is better than complex.
> Complex is better than complicated.
> Flat is better than nested.
> Sparse is better than dense.
> Readability counts.
> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
> Errors should never pass silently.
> Unless explicitly silenced.
> In the face of ambiguity, refuse the temptation to guess.
> There should be one-- and preferably only one --obvious way to do it.
> Although that way may not be obvious at first unless you're Dutch.
> Now is better than never.
> Although never is often better than right now.
> If the implementation is hard to explain, it's a bad idea.
> If the implementation is easy to explain, it may be a good idea.
> Namespaces are one honking great idea -- let's do more of those!
So things like readability beats conciseness, so a huuuuge complex list expression is less pythonic than a manual loop when it becomes too complex.
That is actually a pretty good example of stuff that - in my opinion - hinder readability. I prefer any given day a if <something> assign; instead of assign if <something>. Maybe its because most languages I work/worked with works this way, maybe its because it forces you to read the whole line to actually understand there's an if condition. If I wanted to work with semantic english, I'd use cobol instead.
>For instance, because of the important whitespace, and no end brackets or `end` keywords or whatever at the end of `if` or function definitions, large nesting inside functions becomes very klunky to follow the logic. So idiomatic would be to use small named functions, and split things into smaller pieces once they become too nested.
That's not idiomatic, that is common sense, and common to basically every language. As a rule of thumb, if you have nestings grow deeper than 3-4 levels you're probably better breaking it. This is also a common cyclomatic complexity measure in most static analysis tools. Python makes following the logic clunky even in one liners, because it isn't obvious your code is not branchless. I also understand it is a matter of taste, but I do get quite annoyed with the whole "idiomatic python" thing, because often is just stuff that encourages what - in my humble opinion - are often less-than-ideal practices, that don't translate well to other languages.
> So things like readability beats conciseness, so a huuuuge complex list expression is less pythonic than a manual loop when it becomes too complex.
You don't need to have a huuge complex list. Imagine the example you gave, where later you need to add 2 more conditions to the if clause. What approach you think will age better between changes and refactoring? Might as well do it without list expressions from the get-go, and not having the novice developer who will be maintaining it have to read it 2 or 3 times to understand what is going on.
There's a sentiment that design patterns are ways to make up for missing features in a language. I think that misses the forest for the trees as when people say that they almost always mean the GoF patterns, and not the concept of patterns in general.
Strategy pattern is a great example. First class functions make the GoF pattern a moot point. But it's still useful to understand that this is a useful thing to want to do, whether or not one has direct language support.
I've come to think that that perspective unnecessarily conflates the design pattern with the strategy used to implement it.
Things like the GoF model for how to implement Strategy are kind of a big deal in languages that lack first class functions. But there is also universal value in the basic idea of structuring your code so that you can inject behavior from outside. It's so valuable that many languages weave it into the very fabric of their design. FP fans fixating on the complexity of things like the GoF Strategy pattern and concluding that design patterns are just a band-aid over missing language features strikes me as being like fish noticing the inconvenience of humans needing plumbing and drinking vessels to get their water, and congratulating themselves for not needing water.
This is exactly what I'm saying. It's often true that for the specific GoF implementations they're often a band-aid. But what you said is also true, and much more important.
Better languages change the way we think. First class functions make passing code around so easy and natural there's barely even a name for such a thing. It's just something you do without even thinking. In lesser languages, passing some function around is such a herculean effort people had to ritualize it into an accepted convention with a name and everything.
No matter what pattern we look at, we find it's actually hiding some absurd language weakness. Factories? They're just normal methods that hide the new keyword which introduces hard coupling at the ABI level and can't construct subtypes. Singletons? They exist to work around the fact that classes don't act like objects: classes with static state are natural singletons but they can't implement interfaces or be passed around as objects.
In better languages such as Ruby we discover that new itself is actually a factory method and nobody even realizes it. In Ruby classes are objects and natural singletons and there are no limitations to work around.
And then this one digs more specifically into how not to write Python like it's Java, starting at 12:40 (with talking about why leading up to that): https://youtu.be/wf-BqAjZb8M
I'm confused by the Borg pattern. It seems like a singleton that tricks the user into believing there are multiple instances (very bad imo). Is there any advantage over singleton? Maybe some kind of `__dict__` behavior I'm not aware of?
I showed up 30 seconds after you posted your comment to say the exact same thing. It’s funny that the catchy name was the first one we both went and looked at.
The thing is, this pattern is crucial in C++. It’s how MLIR works. All objects are actually values that have shared internal state. Meaning you can copy them around as much as you want, just like Python, and you’re not copying anything except an internal pointer. It’s like a smart pointer but without any pointer interface, and I’ve wished everything in C++ worked that way. (Memory is scoped to nearest enclosing context, and contexts manage the objects below it. Very simple.)
The other case that this is useful is when you have a bunch of views into some data. Consider the concept of a global variable. Globals are great. I love them. But they require discipline. If you want to spin up a bunch of threads that each have their own view of that global, you’re hosed.
Except you’re not. What you can do is have a thread local variable that initializes itself to the value of the global variable. That way new threads start with the current value of the global. Why? Because suddenly you can just pretend like you’re using globals everywhere! Whenever you want to dish out some work, set the global to foo, then spin up a thread. Set the global to bar, then spin up another thread.
Instead of passing that
damn value through 100 smaller functions, all the code simply refers to “the current user” or “the current webpage” or “the current widget”, which is a global concept. If you ever need to modify it, you set the global and spin up another thread.
Internally it’s implemented by a pattern similar to borg. The instance returns its thread local state, unless it’s nil, in which case it initializes from the global value.
The nitpick is that this needs to be true for child threads that spawn threads, otherwise the whole thing doesn’t work well anymore. Racket nailed this concept with thread cells. If a thread cell is preserved, any child thread immediately inherits the value from the spawning thread. And it’s all transparent.
With this borg example, it’s just literally a global variable. Or maybe it’s not and someone will come say why that’s mistaken. But I don’t think so!
It's basically a global / hoist with a more ergonomic way of accessing the state. Instead of a get_global_state() style function, you just have the Borg constructor.
Personally I'm a fan of dependency injection whenever possible. But borg pattern has gotten me out of a pinch (exactly) once.
My understanding is that Borgs can be subclassed, you can modify the behavior, use descriptors (setters/getters), etc. You also avoid the `global` keyword.
It's really not the most compelling reason. I recently found a cleaner way that actually achieves identity-equality, which I think is a much better way of going about things. Not actually using it anywhere at the moment, but I think it's neat.
class Singleton(type):
def __init__(cls, name, bases, dict):
super(Singleton, cls).__init__(name, bases, dict)
cls.instance = None
def __call__(cls, *args, **kw):
if cls.instance is None:
cls.instance = super(Singleton, cls).__call__(*args, **kw)
return cls.instance
class MyClass(metaclass=Singleton):
...
class MySubclass(MyClass):
...
>>> MyClass() is MyClass()
True
>>> MySubclass() is MySubclass()
True
>>> MySubclass() is MyClass()
False
I've had Python code reviewed in the past by senior devs from more of a Java/C# background, who encouraged me to read the GoF and rewrite classes using the abc (abstract base class) library, staticmethod/classmethod decorators, etc. I never quite understood the value of it in Python. But the fact that those are included in the stdlib makes me think that they must have some clear use cases.
Picking very specifically on the builder pattern: the whole point is to solve one or more of the following problems:
1. creating complex objects and having them valid at creation
2. many optional arguments for an object's construction
3. similar types for object construction
For e.g. in Java you might see something like:
Point p = new Point(10, 0, 5); // (x=10, z=5)
Point q = new Point(10, 2); // (x=10, y=2)
which may be problematic because
1. y needs to be specified explicitly to 0, because you can't expose an overloaded constructor that takes x and z (that would clash with the constructor that takes x and y
2. It _may_ not be evident that new Point(10, 0, 5); passes x, y and z in that order
3. It ties the constructor to the implementation. You can never expose a Point(double radius, double theta) constructor because that would clash with the Point(double x, double y) constructor.
The builder would solve all of these problems:
Point p = Point.newBuilder()
.setX(10)
.setZ(5)
.build();
Python however doesn't necessarily need this pattern because idiomatically solves all three of these problems by providing
1. named arguments, and
2. default arguments
so you could have:
p = Point(x=10, z=5)
q = Point(10, 2)
t = Point(radius=5, theta=45)
I feel the post completely glosses over this by providing a single super-class with the appropriate constructor, and overriding the methods in the sub-classes. What if I just have a single class that I want a builder for? I don't think it showcases the builder pattern at all, just method overloading in classes.
I recently began reading GOF (a bit into the behavioral patterns at the moment). A lot of those patterns were ideas I had been familiar with, but not through a formal specification, likely due to seeing so much code inspired by the book. I just find it so fascinating that these patterns were written about in 1994 (?) and they were pretty well designed.
I will say, the book is heavily focused on OOP, which has fallen out of favor a bit with newer languages (but obviously very alive), and it's almost like a time piece. Feels like OOP was seen as this answer to so many business problems and organization of code.
Just interesting to imagine OOP at a time when it was new and exciting, since I grew up at a time when Java was ubiquitous and everyone was at least a little OO.
You want something that encapsulates state and a method. That's precisely what classes are for! I feel going against how the language is meant to so is just going to make things more difficult for yourself.
Is using modules like that nicer? That's subjective, I guess. I don't think it's nicer. You could use dataclasses to make classes somewhat nicer (though that is subjective too).
It it leaner than classes? I don't think it's something to worry about. Maybe use __slots__ in your class if and where needed. Is an encapsulated module still leaner than a class?
In the end both modules and classes are namespaces. Modules are singletons, classes have as many instances as you need.
Or else a class. I don't quite understand why you'd want a global in a module not mean the canonical 'this singletone thingie configures this module' ?
I don't understand, this looks like a class? Put all that in a class called Greeter in greeter.py and call it a day. Anything beyond that, I'd question very heavily "Why?" to justify the complexity/non-standardness.
>> Do you think it negatively affects performance?
Yes and no. Pragmatically, no, because in python usually performance comes from (ignoring using native code) avoiding dynamic dispatches (circa 500x speedups are common) and like in almost all languages, being aware of your allocations (varies wildly but avoiding allocs in loops for example will give remarkable speedups).
In theory though, yes absolutely this is slower - what was a query of a dict (or even better, an array in the __slots__ case) - i.e. grabbing an attribute from a class - is now going outside the process and asking the OS about stuff on the filesystem. Masssively slower.
>> Why don't you think it's a good idea?
Because you’re paying an overhead to use something in an unexpected way when the language provides a faster, idiomatic solution to this problem:
>> a pattern to encapsulate data and functions
That’s the class and object system described in one sentence
>> I don't see how __slots__ would be relevant to this?
You’ve heard of turtles all the way down? In python it’s dicts all the way down. Behind the scenes you can think of a loaded module being a dict, just the same as an object has a dict underpinning it.
Slots comes in as an optimisation on this dicts all the way down. An optimisation that’s not open to you if you (ab?)use modules like this.
But i’d finish up by saying this - i’m a random person on the internet that doesn’t know your use case. I stand by the above as decent general case advice but you might actually have a valid case to deviate AND you’re not going to unlock new discoveries in better ways to do things if you just always follow the dogma so don’t be put off by my negativity toward the idea but DO measure the outcomes you care about when experimenting to see if the modules idea is working for you.
If I understand you correctly, only the creation of greet1 and greet2 are slower, but then the usage of these objects is just as fast as using instances of classes?
I don't think you want this, but just in case you do :)
def encapsulate(mod):
import types
out = types.SimpleNamespace()
def replace_global_scope(f):
# via https://stackoverflow.com/a/1144561, but tweaked for py3
return types.FunctionType(
f.__code__,
out.__dict__,
f.__name__,
f.__defaults__,
f.__closure__,
)
for name in dir(mod):
val = getattr(mod, name)
if callable(val):
val = replace_global_scope(val)
setattr(out, name, val)
return out
I'm having trouble determining if this is parody. It appears that you want modules to replace classes, and also for users to roll their own __init__ methods? I guess that's "lean" in the sense of malnutrition, but I don't know about "nice."
How would encapsulate look like? Would it have a negative performance impact?
I don't care about code completion at all. When I type "a", I don't want anything to happen on my screen except for an "a" to appear. I think that all the flickering stuff that the average programmer experiences on their screen burns more mental energy than it saves.
You could normally do this with copy.copy or copy.deepcopy but apparently you can't pickle a module object... A quick internet search does not immediately reveal anybody trying to do this before, and I am surprised.
Not with the standard library pickle, but "dill extends python’s pickle module for serializing and de-serializing python objects to the majority of the built-in python types".
Dependency injection and lazy_property are excellent, I use those all the time. I've used visitor on occasion. The rest? Mostly meh, some are straight up bad.
I have found that Python really can show you what other language is someone's primary language. You can write Python to look VERY Java-like, or very Ruby-like, or many others.
Idiomatic Python does not look like this generally. Another commenter hit it well that this is just badly-hidden Java code written with Python.
Python is what you make it to be; don't let this kind of stuff from random people on the internet scare you into thinking you have to use GoF and Java-like patterns and classes everywhere. You CAN, but only if those are actually _useful_ to you.
The stuff that is actually idiomatic python is either too convoluted or rather obvious.
I know this comes across rather mean, but this is just not a good resource, for either beginners (not very idiomatic) or experienced developers (trivial patterns with trivial examples). There's maybe 2 or 3 good ideas in there, but even those I might be hard pressed to find a real usecase for.