High. This past winter here was one of the coldest ones in more than a decade. South Africa is also the only country in Africa(south of the Sahara) that really tests.
A single core running Python will easily handle more load than most successful SaaS companies ever reach, especially in web dev where most of the real work happens in a database of some flavor.
There absolutely are performance pitfalls, and Python is less power efficient than other alternatives running some kinds of workloads at scale, but in a ton of environments the perf difference doesn't matter.
But while I too had in my head "for web dev most of the real work happens in a database of some flavor", I recently realized that was not true for my app, probably hadn't been true for some time, and probably isn't true of most Rails apps (possibly not the same for non-Rails web apps?). If you've properly eliminated n+1 queries and other inefficient querying, I find that my apps are spending only 20 or 30ms waiting on DB results, and a couple hundred on CPU tasks to render HTML.
I know people are going to reply with "That's because Rails is slow," but I'm not sure that's true for what we're talking about compared to similar Python platforms, I'd be interested in seeing numbers for other real apps. In the Rails community too, people still repeat the assumption "most of your response time is spent waiting on the DB" -- but I dont' think it's actually true anymore (it may have been once).
The "most of your response time is spent waiting on the DB" assumption might not hold for your app. Even if you actively push work to the DB to reduce network bandwidth or something, your workload might fundamentally be comprised of small, easy-to-optimize units of work that the DB handles without any issues.
That said, hundreds of milliseconds sounds slow by an order of magnitude or more for html rendering, even if all the work is done in a batteries-included framework for a dynamic language, and it doesn't jive with my Python experience at all. Do you mind me asking what kinds of tasks are taking that much time?
In this app, mostly lots and lots of thumbnails. It may be some unoptimized code.
But rather than get into the details of my app, and parts that need optimization (whether in my local code or in Rails or in ruby), I'm more curious about the overall concept.
Are you sure that most of your web app's time is spent waiting on the DB? My suspicion has become that this is conventional wisdom that is not actually true of most apps anymore. But I could be wrong. Or I could be right only for Rails and not python because Rails is slower than typical python, or something. I am curious to find out.
> Are you sure that most of your web app's time is spent waiting on the DB?
Positive. It's my first tech job though, and we use C# at work to the extent that matters.
The kind of Python I write off the clock probably isn't a good example of waiting on the DB though. It's usually a thin wrapper around C or assembly (or calling into such a library via numpy, networkx, etc) to do something horrendously expensive that I absolutely would not want to run in vanilla cpython. That said, when I do ordinaryish web-related stuff in Python I'm looking at well under 20ms total elapsed time per call, which is why your ~200ms html rendering time stuck out to me.
time "spent" in the database depends on the speed of DB drivers too, which depends on the language. Our stuff is Java and average query is ~2ms round trip.
Python, Ruby and friends are bad for page speed metrics. REST response time of ~200ms vs ~10ms for our java backends
Yes, it does, as long as performance-bottlenecked code is not in Python (and it usually isn't). In the real world, you may have to extract some hot regions to C++ or Rust or Java, but 99% of your code will scale infinitely.
My team migrated a service handling >1.6k req/s at peak time, from Scala (Finatra), to Python (Django). Same number of servers (3x c5.large), just higher CPU utilization (from ~15% to 70%).
Scala was great, but unfortunately it made it difficult to onboard new team members into the project, so development suffered. Also the rest of our stack is mostly Python, so we couldn't use a lot of the common tooling, and libraries we had built for other projects.
We migrated it within a month, and it's been running for almost a year without issues.
At some point the service even recorded 3k+ req/s at ~200ms 99p latency. Yeah, maybe not Google scale, but more than enough for 99% of the businesses out there.
This service handles about 2.5 billion requests per month from one AWS region. We're a small team, and that's only one of the services we support. Which is great because we don't need to spend all of our time optimizing it.
Notwithstanding this (that Python will scale), Python will force you to scale horizontally a lot earlier than something like Rust. In mid-2013, NewsBlur was using 48 servers to run its Python/Django stack. The workload was reasonably precisely declared (and I could grudgingly see why it used so many servers for its stack), so I was able to run the numbers and confidently concluded that something reasonably efficient (which for me would now mean Rust) could have handled the same workload on one equivalent server, though not with a great deal of headroom, and further vertical scaling might have started running into I/O limitations. But for a one-man thing, I know I would prefer to use a single server as long as convenient, and a 98% reduction in hosting bill is very significant (though that’s a moderately extreme case, and I imagine smaller parts of the code could have been rewritten in a language like Rust to get a meaningful fraction of that resource usage reduction).
Please just don't make the mistake that most people I come into contact with make and lump vegans and vegetarians into the same category. Completely different.
This shouldn’t be downvoted. As a vegan it is extremely common to be offered fish or cheese. Travelling in China a few years ago, I ordered through a guide a vegan meal and was surprised to find chicken in it! Best to jettison the terms altogether and focus on Whole Food, Plant Based. Can’t argue fish or cheese or chicken meet that definition!
I saw a poster in an Arby's (fast food restaurant chain specializing in roast beef) that said "Don't Worry, Vegetarians, We Also Have Turkey & Chicken". Here's a photo I found of it [1].
First-class integration in Excel would be a game changer. Holy shit. Though adding a multiline editor with indentation in Excel would be really nice as well. We've got to stop pretending Excel functions don't constitute a programming language.
Bring in versioning/snapshotting with some git integration while you're at it...
The finance folks I work with are fucking wizards. I'm convinced many of them would have made great engineers given what they can accomplish building applications in what might be the world's worst IDE.
Well... most quants are recruited straight out of engineering schools so you are probably right. About the IDE, honestly once you think about it, it actually achieve many thing at once. I am not sure what they could change to make it better.
Some of the most impressive, most complex Excel apps I've seen were 1000% not built from quants out of engineering schools. That's the beauty of Excel - but _managing_ these apps is the nightmare of my specific profession :)
I'd say two things would make it better:
- Have a "code" version of the cell editor that has formatting/indentation + basic linting. [Sometimes 75% of the challenge is reading the code in linear fashion and figuring out what it does]
- Integrate git-like versioning and some sharing features with Office365. Basically, make it so that you can roll-forward or roll-back on both the code and the contents, and make it so that you are _not_ emailing files around to share them.
Interestingly enough, Libre Office (and OpenOffice before that) allow you to use Python internally for Macros through PyUNO, which is a pain to use given the lack of documentation, but amazingly powerful.
It was one of my first usages of Python, and used that to create quite complex Spreadsheets
Office suite automation with Python would be great across the board. The external python libraries that do this are pretty clunky and VB scripting is a hot mess.
I do garlic at commercial scale with fertilizers but my own food is grown 100% organic. There is a big difference in taste. What we should keep in mind though is that organic is difficult/expensive to scale. I doubt that it will be possible to feed the entire global population with 100% organic food. So that tasteless tomato that you buy at the store might not be the best but it does feed you.
Check out "Treating the Farm as an Ecosystem with Gabe Brown" et. al.
He's a commercial farmer that's been doing regenerative agriculture for a while now, he's got "the gift of gab" and his stuff is all scientifically grounded and verified in the real world on his working farm in N. Dakota.
Rather than being difficult or expensive the methods he's using save money, increase the quality and quantity of his topsoil year-on-year, and he actually makes more profit per acre than his neighbors.
> I do garlic at commercial scale with fertilizers but my own food is grown 100% organic. There is a big difference in taste. What we should keep in mind though is that organic is difficult/expensive to scale. I doubt that it will be possible to feed the entire global population with 100% organic food. So that tasteless tomato that you buy at the store might not be the best but it does feed you.
The argument that it doesn't scale negates the majority of Agriculture's 10K+ year history. This modern system of chemically driven commodification of food is rather new (post WWII). Granted the World's population is larger than what it was back then, which introduces challenges, but the truth is we are facing disease due to the over-consumption of the 'abundance' more than anything else in the West, and the East is following as well.
While I agree the current monetary incentives are not aligned for its success, and the business model that had proven profitable for small scale, local and often organic farms working for the past ~15 years has been essentially curtailed due to COVID, as so many restaurants are limited in capacity or shut down altogether, it is not beyond the realm of possibility to transition how we view Agriculture and Food as a whole. I think this re-calibration is necessary for the challneges we face as Species on Earth moving forward and sincerely believe COVID may be the the disruption we needed to question the 'business as usual' model.
There is a great deal of efficacy to the theory/practice that soil remediation to be an effective deterrent in reducing atmospheric CO2 levels; at its core the way Biodynamics has operated since its inception when it was introduced to the nobility in Austria since WWI is nothing more than a distillation of viable practices that follow sound microbiology methods. It may be attached to a great deal esoteric 'woo' but once you see past what is essentially Marketing what you have is the collective synthesis of 10,000+ of trial and error in Man's relationship and the Stewardship we've had with the Nature. That much cannot be disputed.
People not entirely obsessed with this topic forget that a food supply not dependent on inputs like artificial fertilizers, herbicides and pesticides aren't that old and that some countries never adopted it due to the costs associated.
Explaining the significance of this topic without a great deal of context is entirely daunting but, large concentrated populations in former Empires didn't suffer because their wasn't enough BT corn to feed its cattle, and we would be remiss to forget that much.
Detroit's story with urban farming and the culture it brought back with it is a really amazing story that proves that something deemed not 'commercially viable' is not as important as the resilience of a community and a People if the will exists.
> So that tasteless tomato that you buy at the store might not be the best but it does feed you.
'Feed' is questionable, but what it clearly does is introduces a myriad of chemicals into your body that can and has cause(d) other health issues, all while still disappointing your palate.
> large concentrated populations in former Empires didn't suffer because their wasn't enough BT corn to feed its cattle, and we would be remiss to forget that much.
Um, food riots were an absolute MAJOR thing in antiquity and right up until only very recently (<50 years). Urban farming doesn't work at scale. Not sure people realize how much scale modern agriculture has.
> Not sure people realize how much scale modern agriculture has.
I do, I've admitted to be obsessed with the topic and have spent a large part of my life dedicated towards it. Look at my posts to see what I've done.
> Um, food riots were an absolute MAJOR thing in antiquity and right up until only very recently (<50 years).
This is very disingenuous and deflecting comment. What is being argued here isn't that food scarcity didn't take place, for that one really should look towards currency debasement, peasant uprisings, and the clashes in slavery and indentured servitude as well caste systems and feudalism as the likely culprits. Disease and blight both Human as well as crop (like the potato famine) also play a significant part in this story.
But what is being argued is that the advent of CURRENT Ag practices, those reliant on those inputs, aren't commonplace as you are making it out to be in the total History of Agriculture as a whole.
But it must be said that scale means nothing when the most obvious byproduct is disease (heart disease and diabetes kills more than thing else in the West and obesity has been endemic for some time) and destruction of the environment along with it. And that's where we are.
> Urban farming doesn't work at scale.
Urban farming, logistically speaking, is pretty much how most of Europe was built. The wars changed a lot of the landscape of many Capitals and metropolitan areas and displaced many farms outwardly as distributions and supply chains expanded, but in my experience with 15+ European countries I lived/worked in most of the country-side is built in such a way where the cities surround the farms/orchards/vineyards outwardly. They served as a sort of nucleus, which makes sense if you were going design a system to serve a growing population and it's civilization.
The exception being coastal areas that seem to have a more spontaneous pattern about them, likely due to port city trade/imports.
> food supply not dependent on inputs like artificial fertilizers, herbicides and pesticides aren't that old and that some countries never adopted it due to the costs associated.
I was reading a thing the other day (I'll link to it if I can find it) that was pointing out that something like 70% of the people in the world are still fed today by old school farming. Chemical/industrial agriculture only supplies food for about 30% of the world.
> Chemical/industrial agriculture only supplies food for about 30% of the world.
While that is possible, if solely measured by direct to consumer purchases/ I think that may be an understatement as the majority of it is used in the ancillary food system as fodder for livestock (GMO and subsidized corn for the most part).
The numbers are are really hard to get accurately, as the Food Industry as a whole cannot be correctly quantified by its Nature--they span from massive Mega Corp farms and it's food processors to under the radar subsistence local farms and markets that operate on a SystemD model.
I wish it were true, to be honest as it would indicate my revolution/cause has succeeded, my most ambitious target was to get involved and help break into the 50% threshold in my Lifetime: when I started and began this endeavor in my mid teens (the early 2000s) it was nearly ~85% percentage bias towards chemical based conventional Ag.
