> "A program that performs a useful task can (and, arguably, should) be distributed to other scientists, who can then integrate it with their own code. Free software licenses facilitate this type of collaboration, and explicitly encourage individuals to enhance and share their programs. This flexibility and ease of collaborating allows scientists to develop software relatively quickly, so they can spend more time integrating and mining, rather than simply processing, their data."

https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

Now there isn't any area of molecular biology and biochemistry that doesn't have a host of Python libraries available to assist researchers with tasks like designing PCR strategies or searching for nearest matches on up to x-ray crystallography of proteins.

I've been surprised by the rise of Python in some ways although not at all in others. Languages like C, C++, Fortran, and dare I say it Rust are too low-level in their raw state for numerical computing. You had the US federal government funding language competitions because of this (see: Chapel). Languages like Python and R (and before that things like Lisp) came along and gave people a taste of something different, and it's obvious what people migrated to.

Part of it is timing: multivariate computational statistics (ML/data science/DL/whatever you want to call it) just sort of started taking off in computer science communities before LLVM-based languages like Julia or Nim could get a foothold. OCaml might have fit that niche but never got there because of a desire to take a different path, or take the path more slowly.

So people looked for a nice expressive language, found it in Python, and buried all the messy stuff behind wrapper functions and called it a day. It was furthered along by Matlab being another comparison on the other side -- Python looks kludgy compared to modern Fortran or C, but not compared to Matlab.

All that wrapper time in Python has its costs, so I suspect as limits get pushed further we'll eventually see a migration to something else like Julia or Nim, or something else not on anyone's radar.

One moral to this story is that expressiveness matters. People will go out of their way to avoid talking directly to machines at a low level.

I would put it differently. At 30 bugs per kLOC, I'd prefer my codebase expresses a problem & it's solution- and as little below that level as possible.

Each well-vetted layer of abstraction between a scientific programmer and the machine's low level interface eliminates whole classes of bugs that are irrelevant to the problem that user is actually working on.

Where does that number come from?

The important part wasn’t the exact number, but rather the discovery that the ratio is pretty stable.

The thing is, it isn't stable. It just doesn't depend on the language, what is very surprising. But it varies enormously from one study to another and, AFAIK, nobody has a good set of factors explaining it.

At that constant level of effort, we make a certain constant number of mistakes. And that's what I think these studies show.

Personally, I suspect the number of bugs per line is defined by social and psychological factors, and what changes from one language to the other is the amount of effort one has to put into testing and debugging. But well, none of this is obvious to me.

I’m not sure I can agree with this. Both Python and Matlab provide very nice, high level ways to interact with multidimensional data using simple syntax. Under the hood, both will wind up using fast algorithms to implement the operations. C and Fortran require much more low-level considerations like manually managing memory, futzing with pointers or indices, and generally writing a lot more boilerplate code to shuffle data around.

Matlab, despite all its quirks, could probably have won if it was open source. It’s got a very long history of use in scientific computation and a large user base despite its high price.

Matlab also historically was really bad for abstraction and code architecture in general. For example, the hard "1 function per file" rule, which encouraged people to not use functions at all, or if you really had to, write 2 or 3 really huge functions (in separate files). Only in recent years (the past 5 or 10 years) did matlab get OOP stuff (classes) and the option for multiple (private) functions in a single script file (still only one public/exported function is possible per file, because the file name is the function name and matlab uses path-based resolution).

I don’t follow why its array syntax is easier than python though. They’re mostly very similar and the numpy developers seem to come from a Fortran background.

What seems to attract scientists is the REPL and/or notebook UI style/focus of Matlab/Mathematica/Python/Julia/R/... As projects migrate from exploratory to production, optimizing for interactivity becomes a burden -- whether it is Julia Time To First Plots or dynamic typing causing performance and stability/correctness problems in Python code or even just more careful unit tests. They are just very different mindsets - "show me an answer pronto" vs. "more care".

"Gradually typed" systems like Cython or Common Lisp's `declare` can sometimes ease the transition, but often it's a lot of work to move code from everything-is-a-generic-object to articulated types, and often exploratory code written by scientists is...really rough proof of concept stuff.

Julia is the first language I find truly pleasant to use in this domain. I am more than happy to pay a small initial JIT overhead in exchange for code that looks like Ruby but runs 1/2 the speed of decent C++.

Plus, lots of libraries are really high quality and composable. Python has exceptionally good libraries, but they tend to be big monoliths. This makes me feel Julia or something like Julia will win in the long run.

Consider that BLAS written in pure Julia has very decent performance. If you are into numerical computing, you will quickly understand this is crazy.

Carefully written Julia tends to be surprisingly fast. Excessive allocations tend to be a bigger performance problem than raw speed. Of course excessive allocations eventually have an impact on speed as well. There are some idiomatic ways to avoid this.

Being a generalist programmer that sometimes deploys the work that data-scientists craft, I'd really like an environment for this that can compile to a static binary.

Having to compile a whole machine with all the right versions of shared libraries is a terrible experience.

You might be right about the REPL aspect of things. On the other hand, R took off with a pretty minimal REPL, and my first memories of Python didn't involve a REPL. I think as the runtime increases a REPL becomes less relevant, and it seems like most languages with significant numerical use eventually get a REPL/notebook style environment even if it wasn't there initially.

When you are being exploratory even waiting a half second to a few seconds for a build is enough time for many brains to forget aspects/drift from why they pressed ENTER. When you are being careful, it is an acceptable cost for longer term correctness/stability/performance/readability by others. It's the transition from "write once, never think about it again" to "write for posterity, including maybe just oneself"..between "one-liners" and "formatted code". There are many ways to express it, but it underwrites most of the important "contextual optimizations" for users of all these software ecosystems - not just "speed/memory" optimization, but what they have to type/enter/do. It's only technical debt if you keep using it and often you don't know if/when that might happen. Otherewise it's more like "free money".

These mental modes are different enough that linked articles elsewhere here talk about typeA vs typeB data science. The very same person can be in either mode and context switch, but as with anything some people are better at/prefer one vs. the other mode. The population at large is bimodal enough (pun intended) that "hiring" often has no role for someone who can both do high level/science-y stuff and their own low-level support code. I once mentioned this to Travis Oliphant at a lunch and his response was "Yeah..Two different skill sets". It's just a person in the the valley between the two modes (or with coverage of both or able to switch "more easiliy" or "at all"). This is only one of many such valleys, but it's the relevant one for this thread. People in general are drawn away by modes and exemplars and that represents a big portion of "oversimplification in the wild".

This separation is new-ish. At the dawn of computing in the 50s..70s when FORTRAN ruled, to do scientific programming you had to learn to context switch or just be in the low-level work mode. Then computers got a million times faster and it became easier to have specialized roles/exploit more talent and build up ecosystems around that specialization.

FWIW, there was no single cause for Python adoption. I watched it languish through all of the 90s being largely viewed as too risky/illegitimate. Then in the early noughties a bunch of things happened all at once - Google blessing it right as Google itself took off, numpy/f2py/Pyrex/Cython (uniting rather than dividing like the soon after py2/py3 split), a critical mass of libs - not only scipy, but Mercurial etc., latterday deep learning toolkits like tensorflow/pytorch and the surrounding neural net hype, compared to Matlab/etc. generally low cost and simplicity of integration (command, string, file, network, etc. handling as well as graphics output) - right up until dependency graphs "got hard" (which they are now), driving Docker as a near necessity. These all kind of fed off each other in spite of many deep problems/shortcuts with CPython design that will cause trouble forever. So, today Python is a mess and getting worse which is why libs will stay monoliths as the easiest human way to fight the chaos energy.

Nim is not perfect, either. For a practicing scientist, there is probably not yet enough "this is already done for me with usage on StackOverflow as a one-liner", but the science ecosystem is growing [1]. and you can call in/out of Python/R. I mean, research statisticians still tell you that you need R since there is not enough in even Python...All software sucks. Some does suck less, though. I think Nim sucks less, but you should form your own opinions. [2]

[1] https://github.com/scinim/ [2] https://nim-lang.org/

I actually still would prefer Matlab as the syntax is more compact and natural than numpy (which is like a matlabified Python), but that’s probably just due to more experience in Matlab.

Following the licensing and Octave debacle, all my latest code is written in numpy.

If so, I'd consider switching (as Matlab does that better than vanilla numpy). Julia is pretty great in theory. It is still a very new language for my uses, which means the documentation and community are orders of magnitude smaller than Matlab or Numpy/python.

But it looks like they're switching to VSCode as a Julia IDE. Which should give you what you need, you just have to spend some time setting it all up.

https://towardsdatascience.com/how-to-best-use-julia-with-ju...

Looks like Jupyter Lab and Atom/Juno is what I’m looking for. Still not as well-integrated as Matlab is.

I suppose that’s my attraction to Matlab. There’s not a bunch of different programs/environments to juggle to get a good, consistent experience for rapidly developing scientific code (for simulation, modeling, etc). All still possible.

Also, dynamic typing is a boon - and default & keyword arguments is a great feature for complicated, versatile, useful algorithm implementations and interfaces to them. Both of these features have a cost in bigger programs, but they really make Python stand out.

   before LLVM-based languages like Julia or Nim could get a foothold.

Regardless, Arraymancer[2] is quite a gem in the scientific computing ecosystem (doesn't build on nlvm, i.e. the mainline Nim compiler is sufficient).

[1] https://github.com/arnetheduck/nlvm

[2] https://github.com/mratsim/Arraymancer

Yes, the article talks about this: Python is a glue language and the actual heavy duty computation is being done inside an extension module like numpy that's written in a faster language.

Wait, weren't they supposed to be using Ada for everything anyway? What's wrong with Ada?

https://github.com/arnetheduck/nlvm

It's not officially blessed, but it does work.

Python defeated Matlab. I know because I cheered it on. I was there. I watched my roommates and friends struggle with introductory scientific computing in Matlab and I joined the chorus that was practically begging for Python, even though I didn't really like it. I can't even begin to explain how awful it is to try to teach programming concepts in Matlab. But something like Python or Matlab had to be the choice because the schools wanted to teach programming through a language where you could just call "graph" and the computer would display a graph.

Python's team, unlike Lua's, aggressively courted educational institutions by offering scientific, numerical and graphical libraries within a programming language that works like a programming language, not a glorified computer algebra system. They even added a dedicated operator for matrix multiplication. It's a great example of finding a niche and filling it: I still don't like using Python, but I can't dispute that no other language/ecosystem comes close to offering what we need to teach programming to physics students.

You want to beat Python? Build a type system that can capture dimensional analysis. Warning: it won't be easy.

We also ditched simulink because it is very difficult to version control and collaborate with a graphical interface.

Matlab is pushed heavily in the schools so all the engineers knew it and were comfortable with it. Matplotlib and numpy mimicing matlab very closely allowed the transition to be easy. We're not looking back. Only a handful of people still use matlab for their individual work because the python camp hit critical mass and the transition is not hard.

Matlab working to control serial ports, ethernet, visa/gpib instruments, all without the friction of getting extra licenses was icing on the cake. Matlab has a buy the cadillac model: the wheels, doors, hood, gas cap, mirrors are all optional add-ons. Each point causes friction, as only a few people had the whole tool, and therefore nobody could reliably share code.

Curious about your thoughts on pint and Unitful.jl — pint doesn’t really go all the way to a full type system, and Unitful.jl doesn’t work with everything (autograd is a problem still I think). But Unitful.jl is super cool.

https://pint.readthedocs.io/en/stable/wrapping.html#checking...

https://painterqubits.github.io/Unitful.jl/stable/

This is the answer. Scientific Python was originally an alternative to MATLAB. When I was in grad school, I did most of my research in MATLAB. Then we had a visiting student who was doing very similar computations in SciPy, and he assured me performance was not a problem. I migrated my MATLAB scripts to Python and never looked back.

It was only after being a viable alternative to MATLAB did people decide it can be used for much more than what you typically get with MATLAB.

A benefit of Matlab remains that it all comes from one place, with one installer, meaning that you can get a classroom full of students up and running almost instantly. And it offers some relief for students who will never grasp programming, through its collection of pre-written apps.

Hang on! In what world can you just call "graph" in Python and it would display a graph?

In matplotlib on MacOS at least you try that and you get some bizarre shit about how Python isn't a framework, and you google it and find you have to do some obscure import and the import has to be in a particular order relative to other imports (totally unpythonic). https://stackoverflow.com/a/34583958/583763

Jupyter notebook... don't get me started! You do one thing and it starts a "server", and then you use that to start a "kernel" (and if my CS is dodgy and I don't really know wth these things are then I'm not having a great time already). Then this kernel thing is running Python. But oh, what version? And is it using my virtualenv? And then you google some matplotlib imports. And finally, yes you call "graph" and an ugly matplotlib png is displayed rather small in your web browser.

R however. There you just call "graph".

(In particular, that’s how things have gone in the materials physics/solid-state/quantum chemistry field. It absolutely beat out Matlab in other fields. One of the underrated benefits was being a lingua franca across more of physics!)

Yes. I feel like the author conflates the language with the package ecosystem. Pure Python is pretty horrible for scientific computing (3*[3]=[3,3,3] is about as counterproductive to scientific computations as it gets), but Numpy changes the semantics of those operations.

In other words, Python has an absolutely stellar package ecosystem. There have been attempts to bring a package ecosystem to C, but it never took off. However, I do wonder how C would fare if it had.

https://conan.io/

https://vcpkg.io/en/index.html

this implies Python's advantages is not having a package manager, but better teachers or at least teaching better practices, so it isn't even language related.

Most scientists, and their grad students, are trying to do a whole bunch of things in their research, and programming is just one of them. Field work, experiments, data wrangling, writing papers, defending papers, teaching, etc. And most of them do not have access to budgets for programmers or when they do, it's for a limited amount of time and work, meaning they need to be able to pick up and run with whatever the programmer did. So the fact that with Python they and their grad students (who might be there for only 2 years) can be working productively, and figure out what the hell the code did when they come back to it months later, is HUGE. As in, literally blows every other consideration to smithereens. This has meant that over the last 20 years the scientific libraries in Python got mature faster than in any other language, and this in turn has had a snowball effect. And when speed is necessary, C++ extensions can be written. But honestly, most of the time speed is not the main factor.

The downside of Python in my experience is that junior teams can make heinous atrocities when a project gets really big (I have had to step in as CTO to one of those messes, so much as I love Python, I must admit this is true!) But the stuff the scientists are doing is very rarely that big. It's tools programming, scripting, making utilities, data analysis and so on.

Readability counts. In some fields, it counts more than anything. I've worked in about 10 languages now over the last 20 years, and Python is still the easiest to read when you come back to some old code or have to pick up code for a small job, or hand it to a beginner to extend without having them create an unreadable mess. This is what scientists need to do all the time.

Re other people's comments on Python packaging and setup being hard, well honestly I've had just as much pain with Ruby or Node. The shining exception there is R, which is giving Python a run for its money in many scientific areas. R Studio has the best "hit the ground running" experience out there and is really slick for data programming.

In addition to not having budgets for programmers, we also don't know how to manage them, for instance how to communicate our needs, decide if their implementation plans make sense, or gauge their progress. Nearly half a century after The Mythical Man Month, managing software development is still generally acknowledged to be an unsolved problem.

The other two obstacles are that most programmers hate the scientific work environment, with its ever-changing requirements and frequent dead ends. And, the programmers who can work on math related stuff are in the highest demand.

There is no other language I have used that makes it as easy to read code from somebody else, especially where that contributor is likely to be a domain expert with very limited programming experience. It's not actually my favourite language anymore (hello Scheme!) but if you want me to do work in that environment, I'll reach for Python first.

Could you suggest the best way for finding a remote job for such programmer?

I wonder if there's any concerted effort to fix that for academia, or if the "shortage" of math-literate programmers just isn't a problem worth fixing.

I think universities will never be able to compete for engineering talent until they can create attractive career paths for people who aren't professors.

It's also ethically difficult to advocate for higher salaries in the academia, if you are already living a comfortable middle-class life. The money would ultimately come from taxes and tuition fees. If you think that those should be increased, the money would be better spent on helping your colleagues who are earning poverty-level wages.

Engineering is a support role in the academia, because pure engineers don't teach or set research directions. Most labs and most departments are too small to employ more than a handful of engineers, if any. Only large research institutes have enough engineers working on similar topics to justify creating senior engineering roles.

There are people who have stepped off the gravy train because they don't like it, or they don't fit into the enterprise workplace for whatever reason. I might be one of those people. I work in industry, but in an early-stage R&D team.

Maybe the status quo is a reasonable solution: Find grad students who are willing to do the work in return for a chance to sharpen their programming skills. This process could be improved by providing scientists with training on how to write better code. The result will be a certain amount of attrition of scientists into software development jobs, but we have to get used to the idea that attrition into a more employable field is actually a good thing, and there will be plenty of scientists.

I've personally done this by finding normal work that is part time, so I can round it out doing work for scientists, artists, and musicians.

That's what I did for the first 10 years after graduating from university. Eventually I transitioned to a full time 'normal' job but that made me unhappy.

- get more traction

- I'm less able to focus on the software and have to focus on business development

- I'm going to need to hire someone who is a competent programmer but can also deal with the mathy bits

It's pretty fun to do for me, but it's certainly challenging to balance with programming despite my advantage of having taken a lot of extra math classes.

Lucky you. You must not have seen the "pythonic" monstrosities I've seen.

Python has such a low barrier for entry that one can "get stuff done" with absolutely atrocious and often very overly complicated OOP-ish code.

Ruby is not my favorite language, but I would bet real money that without dependence on libraries, nobody could show me Python code which I could not show more logical, consistent, and readable Ruby code which solves the same problem. I say Ruby because it's of the same "type" and follows similar methodologies.

Python suffers from far too many years under the leadership of one odd person. It has a cult-like following, whereby anyone who disagrees is an outcast. Where else could you hear comments like, "why would you ever need a switch statement? if/if else works fine!" That's just the tip of the iceberg.

Python is great for integration glue code, but only because of the libraries it has. But now it is becoming more Javascript like, and the dependencies are multiplying to the point where you're better off writing your own left-pad instead (or even re-evaluating your approach) instead of taking on new duct tape like django-database-view.

Sometimes the bar needs to be high enough to force the juniors to actually learn something before they start building "MVP" startups. On the other hand, who cares if the MVP is a horror show as long as you get that IPO and take your f-u money and leave.

Bad coders can make terrible stuff in any language, and with two as similar as Python and Ruby, the minor differences are a drop in the bucket in the grand scheme of things. Both Django-database code and RoR's Active Record have bogged down many a startup when they got big enough that DB size and query performance mattered.

None of which, as I pointed out, is relevant to the vast majority of scientists writing code.

The takeaway: always, always, have a domain layer that allows you to refactor your model access without changing tons of code. Data load grows in ways people don't predict. If your company succeeds, in five years you'll wish you had it!

Tech debt kills loads and loads of companies and most folks never hear about it because that's not the stuff that gets publicized or written about. We call it the silent killer...

That's really reassuring to hear. Whenever I say we should spend time on tech debt it's always greeted with "we can worry about that later when we're really successful" as if there will suddenly be an opportunity to completely rewrite everything (because that's what it will take).

I keep hearing all the stories about Ruby supposedly being more logical and sound than Python. I really would love to see actual source code being cited to back those claims.

- The OOP features of Ruby are consistent and ubiquitous (everything in Ruby is an object); Python depends on manual patterns to do OOP (self as the required first argument). Python also depends on special decorators to indicate what functions are instance vs static. Ruby does have a difference in definition, but it's simpler and more obvious (and requires one line fewer of code to define)

- Only finally has Python gotten a switch statement, and surprisingly it has adopted some Elixir-ish pattern matching features. Incidentally, some in the Python community are strongly against this new thing. "Why would you need that!?" Prior to 3.10, you would need more complicate if/else if structures in Python to do the same thing you could do in a concise and clear Ruby case (switch).

- Operations on collections: this is often described as functional programming, but it really is just "doing stuff on collections of data". And in that story, Python's list comprehensions are arguably less readable and less logical than Ruby's. Many of the tools you need in Python must be explicitly included from functools module.

- Ternary operator: many languages have `expression ? do_true_path : do_false_path`. It's a very common pattern which is concise and honestly quite clear. "This thing is true? then do this; else do that". But in Python you break that up into "do_true_path if expression else do_false_path".

- Everything in Ruby has a return value, but not so in Python. So in Ruby you can make assignments (or return values) from the result of if/case. For example, assume you want to return a specific value based on some series of conditions, such as handling an error and returning some enriched data based on the error code. In Python you will have to define a local variable and explicitly set that variable equal to some value in each branch of the conditional. Then afterward, you can use the value of the local variable. Or you would have multiple returns, one in each conditional branch. In Ruby you can simply do x = case ..., or because the last statement of a function is the return value, you wouldn't even have to return it. You just have 'case ...', and the value of the branch is what is returned.

There's a lot more. Some of it is subjective, but my belief is that once someone really knows both, they will prefer the Ruby way. And the more languages you know, the more you develop refined tastes. Ruby still holds up well after knowing 10 languages for me.

(added): the whole whitespace as code thing of Python. It does have one common pitfall, and that is when a line of code gets accidentally indented or unindented below a block which was indented. That line changes scope, likely changing the runtime result; but it may be technically valid, so the developer may not notice the mistake. This is just not a problem with languages that have { } or begin/end delimiters.

The significant dangers in Ruby lie in the metaprogramming, which probably scientist devs aren't exercising.

Meh. Python might be readable at the smallest scale, but then COBOL is even more readable. What matters is large-scale development, and your implied point that large Python projects turn into unstructured big-ball-of-mud monstrosities is well taken. A big ball of mud is not surveyable, or "readable".

Which is where other modern languages (e.g. Julia in the scientific programming domain, heck even Go or Rust) will probably have an advantage.

I like all kinds of languages, but the only ones I would encourage my partner to bother with as tools for her science work would be R and Python.

At best, they engage in the polite fiction that it just doesn't matter, because all that code is inherently "throwaway" stuff that's only used for playing with in the context of research. Of course even that is wrong, the code doesn't really disappear like that.

I would not be patronizing to these people, they are very smart. They just live and work in a world that is completely different from technical product firms in pretty much every regard.

Not for the kind of computing being talked about.

You nailed it.

I suspect MicroPython is going to do the same thing to Arduino/C as Python just did here in Academia as well.

I certainly think that MicroPython serves a niche, primarily very simple hobbyist/educational roles. However, I do not regard it as suitable for anything beyond this. It's the wrong tool for the job, and if you want a scripting language for low-latency low-overhead use, there are smaller and more efficient languages which fit better into an embedded role.

Bioscience / environmental science programs will find micropython good enough for their needs. The tool itself is just the means to the end if real science. Micropython let’s you deploy in lower power applications without having to learn much beyond what you already know from Jupyter notebooks.

I really don’t know any PhD students or post-docs in microbiology/environmental sciences who have the time to learn embedded C or similar languages.

> As the name suggests, numeric data is manipulated through this package, not in plain Python, and behind the scenes all the heavy lifting is done by C/C++ or Fortran compiled routines.

So I wonder, was it easier to write C/C++ or fortran compiled extensions in python than it was in ruby?

Ruby is very productive to write, because everything and the kitchen sink is at your fingertips at all times.

But because of Ruby's many ways to skin a cat, everyone's code is very different. Add to that the penchant for domain-specific sub-languages in Ruby: new syntaxes that you might have to learn half a dozen of to integrate a large project, all of which end up being more limiting than if you could just, you know, write Ruby.

Contrast with Python, which goes so far at normalizing as to have a language-wide coding standard in PEP8. Python has its problems, package management and distribution is still ugly for example. But I can read any project I find and understand it without loads of context.

Even proponents will say things like "this language is so expressive, I feel so productive in it, but people do such idiosyncratic and clever things that it's hard for anyone else but the author to understand".

That sort of "solo rockstar" programming culture doesn't really lend itself to large-scale FOSS projects, which need to be inviting to wide participation.

For large projects with multiple developers, readability should win over writability every time, most code are read more than they are written (my hypothesis). You can see evidence of this, given the success of languages like Python and Go.

That said, for scientific compute, in a lot of cases, writability matters way more, as your job as a scientist is to produce results as fast as possible, code quality be damned. However, only a small number of scientists are expert developers and have climb the learning curve and can write code with ease. The vast majority of them are junior at best, and Python's approachability (which is rooted from its readability of course), wins. With most of the people using Python, the ecosystem develops and there are no other viable alternatives. In the long run, I suspect even languages like MATLAB and Mathematica will die out as the open source stack becomes more mature and (eventually, if not already) significantly more capable. Julia might be a wildcard due to its (potential) performance advantages, but the aesthetics of the programming language is simply not in the minds of 99% of the scientific compute users out there.

If it's just people who have done a lot of work in ruby and little in python saying ruby is more readable, and vice versa, I don't find it very useful even anecdotally.

I suspect the main reasons are:

1. Python's guiding philosophy of "There should be one-- and preferably only one --obvious way to do it". With later additions to the language, this is getting less true (3 different ways to format strings, asyncio, type hinting, etc). Some libraries also don't conform to this (matplotlib). That said, it's a lot better than the Ruby code I've encountered, which is like the wild west.

2. Python's syntax is reasonably simple to teach. The object model could be condensed into something very simple if you don't need a lot. With very basic knowledge, you can go a long way. Ruby's a bit more chaotic with things like inheritance, extend, and include; proc, block, lambda; having to use attr_accessor; syntax things like a = b could be a function call or not; if/unless; and many more things that are confusing.

3. Even basic things like loops in Ruby is not idiomatic as it wants to apply a function/block instead. Beginners, especially those with a bit of background, like their loops better than functional programming.

As I've spent many years working on Ruby code base I still get lost all the time. Python in my experience has been a lot better, although recent Python versions have regressed a bit as it introduced more syntax to do the same things.

Don't know about technical aspects of "easier" but it may have simply been an accident of history.

E.g. in 1995 (before Ruby 1.0 December 1996[0]), David Beazley was already wrapping C Language code to Python. Deep link to presentation: https://youtu.be/riuyDEHxeEo?t=52m27s

So DB's Python code for scientific code was released in Feb 1996 and presented in July 1996. Python being released in 1991 was already talked about in magazines in 1995. David's presentation also references Jim Hugunin[1] and he authored the 1995 Numeric package which was the ancestor to NumPy. Once an ecosystem gets started, it can attract more mindshare and snowball into an insurmountable lead that neither Ruby nor Julia will ever catch up to.

In other words... If the opposite timeline happened and Ruby was released earlier in 1991 and Python later in 1996, things may have played out differently.

So folks like David Beazley and Jim Hugunin chose Python as the scripting host language for their C Language code probably because Ruby wasn't mature and well-known back in 1995. Apparently, Ruby didn't widely spread outside of Japan until 1998 when the first documentation in English appeared.[2]

[1] https://youtu.be/riuyDEHxeEo?t=30m04s

[2] http://blog.nicksieger.com/articles/2006/10/20/rubyconf-hist...

It was clear at the time that there basically was no non-web Ruby community. Ruby was Rails and Rails was Ruby. Ruby had a nice little niche in 2009 but the Python had Numpy and there were a lot of ML people doing lots of math and Ruby wasn’t going to cut it unless they wrote their own libraries, which wasn’t worth the effort since Python and Nunpy already existed and already had a growing community behind it.

Numpy.

I honestly think it all boils down to numpy being developed long before matrix libraries became a standard part of software development.

Ruby's early "killer app" (remember that term?) was Rails. Even to this day there is almost no major code out there built in Ruby that isn't ultimately related to building CRUD web apps. While Ruby may be losing popularity now, it moved the web-development ecosystem ahead in the same way that Python has moved the scientific computing world ahead.

20 years ago if you wanted to use open source tools to performant vector code there was Python and a hand full of oss clones of commercial products. Given the Python was also useful for other programming tasks in a way that say Matlab/Octave is not, it was the choice for more sophisticated programmers who wanted an OSS solution and need to do scientific computing. This creates a positive feed back that persists to this day.

Given that Python remains a decent language relative to it's contemporary peers and it has a massive and still growing library of numerical computing software it is extremely unlikely to be dethroned, even by promising new languages like Julia.

Even to this day there is nothing even close to numpy in Ruby. I do DS work in an org that is almost entirely Ruby, but we still use python without question because we know re-implementing all of our numeric code into Ruby would be a fools errand.

Had ruby had early support of matrix math, it wouldn't have surprised me if it would have replaced Python.

But that begs the question -- why did numpy develop in python and not ruby?

The rest of the thread offers some suggestions though. One is simply that python was born first, and got the numpy precursor before ruby 1.0 even happened. Which seems like a thing.

Lots of comments in this thread from people who's Ruby experience is only from the post Rails era after ~2008, and don't understand that the post Rails culture wasn't really a thing when Python was first gaining momentum for scientific computing.

1. pairs with jaimebuelta's artistic vs engineering dichotomy, but also plays into the scientist wearing many more hats than just programmer. Code can be two or more degrees removed from the published paper -- code isn't the passion. There isn't reason, time, or motivation to think deeply about syntax.

2. For a lot of academic work, the programming language is primarily an interface to an advanced plotting calculator. Or at least that's how I think about the popularity of SPSS and Stata. Ipython and then jupyter made this easy for python.

For what it's worth, the lab I work for is mostly using shell, R, matlab, and tiny bit of python. For numerical analysis, I like R the best. It has a leg up on the interactive interface and feels more flexible than the other two. R also has better stats libraries. But when we need to interact with external services or file formats, python is the place to look (why PyPI beat out CPAN is similar question).

Total aside: Perl's built in regexp syntax is amazing and a thing I reach for often, but regular expressions as a DSL are supported almost everywhere (like using languages other than shell to launch programs and pipes -- totally fine but misses all the ergonomics of using the right tool for the job). It'd love to explore APL as an analogous numerical DSL across scripting languages. APL.jl [0] and, less practically april[1], are exciting.

[0] https://github.com/shashi/APL.jl [1] https://github.com/phantomics/april

In contrast, I've never really encountered anyone advocating for Ruby outside web development.

While van Rossum drew from that experience when making Python, the initial driving goal was as a scripting language for system admin tasks in the Amoeba distributed operating system.

https://docs.python.org/3/faq/general.html#why-was-python-cr...

David Beazley talks about this in a YouTube video somewhere. (Can't find it right now, maybe someone will in the comments.)

It was a lot of serendipity. Python was up and running when the US national labs wanted to collaborate and their tools all sucked. Since they wanted visualization this left only Tcl/Tk or Python/Tk. And Beazley was hanging around as a grad student in a national lab with a connection machine, no real boss, no real oversight, and very little budget. He built stuff out of Python, and it snowballed to other labs.

(Found it: see jasode's response) https://www.youtube.com/watch?v=riuyDEHxeEo&t=1804s

However it is very very easy to write Python bindings for a C/C++ library with minimal work. Solutions range from "just works" like ctypes to "actually integrates with the language" like Cython. You also have automated tools for wrapping like pybind11 which does a lot of the heavy lifting for you.

Python was already taking over in many use cases by late 2000s.

Ruby was known, but it didn’t have the following at multiple levels in academia like Python did

That's the question, not the answer!

It seems like a lot of the answer is NumPy, which makes the question -- why did NumPy happen on python, not ruby?

Certainly one answer could be "nothing having to do with the features of the language, it's just a coincidence, they chose to write it in Python, if those working on numpy had chosen to use ruby instead, history would be different."

But one hypothesis is that maybe NumPy wouldn't have been as easy in ruby as python.

Someone else suggested the first numpy release happened before the first ruby release, so that could also be an answer.

People attracted to Ruby are mostly of an "artistic mindset", they want to be expressive, write code that doesn't look like programming code and using "magic" like dynamically created methods, monkey-patching, etc is accepted or even encouraged.

On the other hand, Python attracts more people with "engineering mindset", they like straight forward code that's readable, clear and understandable, even if it's not as expressive. "Magic" elements are frowned upon: for example, imports are explicit and always included in each file.

Obviously, I'm exaggerating it, but I think is a clear differentiation between the communities.

My guess is that the "Python mindset" got into creating better integrations for "engineering applications", like NumPy or SciPy, and that created some positive feedback in certain environments. The main strength of Python is its rich ecosystem of third party packages. There's a compounding effect, making it grow faster and faster.

Ruby makes it easy to do "magic". Which is fun to write, but painful to read for others.

I've encountered real cases of ruby code where a simple code snippet behaves differently with and without a `require` (IIRC, some utility function added to a class with monkey patching). In another case I've also had to modify (and to some extent maintain) a codebase that relied on overriding `method_missing` in the happy case / normal flow. I was trying to find out where some method was being defined by grepping the whole codebase. It probably cost me half a day of unabridged profanity.

In theory you can do the same thing with python -- thing is it usually doesn't happen for some reason (likely the ones you mentioned). Something about the language features and the culture in the community lead to devs doing different things with the different languages. But the effect is real, and I know which language to avoid if I had the choice.

Around 1993 I got hooked on Perl. I read the Perl book and it was great. But 1) I couldn't figure out how to handle complex data structures (this was Perl 4), and 2) I couldn't embed it into other projects.

More specifically, worked on a molecular visualization program called VMD. It had its own scripting language. I wanted a language to embed in VMD that was usable by my grad student users. This is when I first learned about Python, but I chose Tcl because it fit the existing command language almost perfectly.

At around the same time, UCSF started embedding Python for their molecular visualization package, Chimera, so it was already making in-roads in structural biology.

I later (1997) went into more bioinformatics-oriented work, where I did a lot of Perl. I tried out one implementation (a Prosite pattern matcher) in Perl - which took me reading an advanced Perl book to learn how Perl 5 objects worked. I then tried the same in Python, a language I wasn't as familiar with. And it was just so much easier!

At this time Perl was THE language for bioinformatics, but I thought it was a difficult language for complex data structures. (Bioinformatics at that time was mostly string related, plus CGI and databases - Perl was a great fit.)

I then moved over (1998) to cheminformatics, working more directly on molecular graphs. Python was a much better fit for those data structures than Perl. I started using Python full-time, and it's been that way since.

We used a third-party commercial package for the underlying cheminformatics called the Daylight toolkit. It had C and Fortran bindings. Someone else had already written the SWIG configuration to generate Perl, Python, and Tcl bindings, but these still meant manual garbage collection.

I was able to use __getattr__, __setattr__, and __del__ to turn these into a natural-feeling high-level API, hooked into (C)Python's reference-counted garbage collector.

I presented a couple of talks about this work, got an article in Dr. Dobb's (!) and got consulting work helping companies which either had existing Python work, or were moving to Python.

By contrast, I don't think I heard about Ruby until 2000 or so, years after Python started entering structural biology/cheminformatics. [1]

I wasn't particularly cutting edge - others had already developed tool like SWIG, which was because Beazley and others were using Python at LANL. Numeric Python started in part because of work at LLNL and other research organizations. The concept already firmly established was that Python would be used to "steer" a high-performance kernel.

And Python in turn changed, to better reflect the needs of numeric computing, in particular, the "..." notation in array slices was added to make matrix operations easier. (This was 20 years before '@@' was added to simplify matrix multiplication.) I believe the needs of numeric computing also influenced the changed to "rich" comparisons.

This all took place around the time Matz started developing Ruby. Python had a clear head-start. And except for bioinformatics, Perl never had much presence in the fields I worked in.

So:

> why did python's use in scientific computing keep expanding, and not ruby's?

Because Python was in-use several years before Ruby, and already rather visible as one of the three main languages to consider in that space (Tcl and Perl being the other two).

> Why was python already taing over many use cases by the 2000s, but not ruby?

Because people didn't really know about Ruby, while Python already had a pretty large user community. Probably also because Python's work was all in English, while a lot of the Ruby community was using Japanese.

> Why is Perl attrition relevant, when ruby was in fact explicitly based on Perl?

Perl attrition started before Ruby was much known. The complexity of the language, and the cumbersome need to roll-your-own OO, made it difficult for me to recommend to the typical software developers I work with - grad students and researchers in the physical sciences with little formal training in CS. Python by comparison which easier to pick.

So a language which explicitly based on Perl also picks up that negative impression.

(FWIW, I think Tcl is an easier language to start with than Python.)

> why did NumPy happen on python, not ruby?

Numeric computing in Python started before Ruby was much known. Quoting https://en.wikipedia.org/wiki/NumPy

"""In 1995 the special interest group (SIG) matrix-sig was founded with the aim of defining an array computing package; among its members was Python designer and maintainer Guido van Rossum, who extended Python's syntax (in particular the indexing syntax[8]) to make array computing easier."""

Quoting https://en.wikipedia.org/wiki/Ruby_(programming_language)

"""The first public release of Ruby 0.95 was announced on Japanese domestic newsgroups on December 21, 1995. ... In 1997, the first article about Ruby was published on the Web. ... In 1999, the first English language mailing list ruby-talk began, which signaled a growing interest in the language outside Japan."""

[1] Ha! I found a comment I made in 2003 saying I had looked into Ruby "a few years ago", at https://groups.google.com/g/comp.lang.python/c/xBWUWWWV5RE/m... . I also wrote:

""" I think my criteria for selecting Python over Perl is still true for Python over Ruby, in that it has too many special characters (like @ and the built-in regexpes), features (like continuations and code blocks) which are hard to explain well (I didn't understand continuations until the Houston IPC), and 'best practices' (like modifying base classes like strings and numbers) which aren't appropriate for large-scale software development."""

One of Python's original use cases was as a macro/script language you could import into your C application. Adding python to your C app took a day or so, and a side effect often was you'd make a Python library out of your app's library and suddenly, you could write standalone Python that called your app's code. Because it was so easy to write a python wrapper for an existing C library, by 1995/96 when Ruby hit the scene, Python already had quite a bit of importable functionality. The first serious web framework for Python was Zope, and it came out in 1998. I think Rails was around 2005/2006, and it was really cool, but one of the rubs was Ruby didn't have the libraries that Python did. In reality, it's amazing how good Ruby and Python have been.

Work on numerical packages and scientific computing started almost as soon as the language did, for instance the origins of Numpy lie in the Numeric package which was introduced in 1995.

And the core team introduced several niceties at the behest of the scientific community (advanced slicing for instance, more recently the matmul operator).

i discovered the language in 98 or 99. it came with some obscure linux distribution and the tkinter module stood out for me. it showed pretty scientific graphs and charts. but the language has to reinvent its community many times since then.

my intuition is that it was popular in europe in the scientific community. not sure i can say the same for ruby.

This has been recognized for some time. The compromise had been to build performance sensitive parts in C/C++ and do the experimental/iteration part in Python.

But today you don’t really have to compromise anymore. We got Julia. It solves the whole problem. You get the interactivity you need combined with the performance.

Of course in my his industry momentum matters. Python has built up the momentum of an oil tanker. Even if you shut off the engines it is going to keep going for many years.

But Julia is the obvious end station. It does all the things HPC and scientific computing needs. But building mains share, documentation, community, polish tools etc will of course take time.

There are other aspects to the "whole problem" -- you also need a massive ecosystem with adoption across disparate communities (devops, web development, etc). And decades of momentum.

That's why Python isn't going away anytime soon, despite its slowness and warts.

Basically modern python offers you a spectrum from easy to understand and quick to write python programs (those will be slow), to purely glue code that connects a lot of high performance c/C++/fortran code. And many scientists will start from pure python code with the help of numpy. In many cases it will be good enough. But if needed you can always interface with other libraries, or write yourself high performance c/c++/fortran code for the most performance critical bit, and use python to glue it together. That flexibility where you can trade speed of writing the code with the speed of execution is very valuable.

If I have a working ecosystem using Python, with students trained in Python, and all previous work in Python, there's a whole lot of opportunity cost associated with me deciding to have the next student use Julia. I'd rather have that student build on existing tools and knowledge and do something new with their time.

There's so many convolved factors here I don't even know where to begin, so I guess I'll just say that I'm glad Julia exists. The author glosses over many decades of programming language and compiler research -- which makes sense, because this is not their specialty. However, what I see is the field of scientific computing migrating from a dinosaur language (Fortran isn't, actually. It just is used this way) and dinosaur practices of writing everything oneself, to one of the slowest interpreted languages that happens to be the most difficult to JIT, and saying this or that about how interpreted languages are slow but library calls are fast to justify this. At the same time they're learning to build a functioning library ecosystem.

Basically, grad students are learning proper programming practices and collaboration after switching to a more expressive language, they just managed to pick the slowest and most difficult to optimize one. Maybe they just managed to wipe some of the slate clean by switching away from Fortran and its culture (the culture being the bad part), and the culture of Python filled the space, creating a net positive but somewhat unfortunate situation.

Just one more time -- the idea that you can call a "faster" language to do the heavy lifting is true of every language and does not justify the choice of Python in particular. The justification for Python is the momentum, and this is in my opinion the only one.

Python is/was chosen because the syntax is clean and expressive and obvious if you speak English (which most people in this context do) and because although performance is almost always worse in interpreted languages, there are clear productivity benefits when doing the kind of programming that is demanded from data science / data processing / etc. Same for dynamic vs static typing and a number of other choices made by Python.

Specifically, many (most?) programs are not long-term maintained projects in this space. A lot of them are just little scripts to convert one dataset into another format, or scrape specific content from somewhere, or support a scientific paper that will not get updated after publication.

But it will take years to build a library ecosystem that can rival the python one.

My personal killer app would be a significantly revamped plotting library/app. While matplotlib is great, it is fundamentally based on imaged-based plotting. The next generation of data visualization, imo, will likely be interactive. Having an interactive plotting library that allows you to produce publication-quality plots faster and simpler (think of all the time spent aligning text manually..) could be a big deal, but it could also not matter as no one else wants the same things I do.

[1] https://github.com/JuliaPlots/Makie.jl

[2] https://github.com)fonsp/Pluto.jl

Ecosystem; mind-share; readability and engineering mind-set; history/Numpy/Matlab; teachability and academic focus.

There are also comments emphasizing the "dynamic" scientific environment and need to just pick up code left by others.

In terms of the latter, could one apparent requirement be this: The main contact should be with top-level code which at least looks like it is interpreted -- even if through compile-with-run-combined and/or memoization? Need part of the user interface, so to speak, be to hide all intermediate artifacts, even the very thought of object code and executables? That such stuff is for, say, "module creators" not primary users?

Personally, I find that many Fortran codes are still used because they have been build for many years, and they can't be rewritten easily. On the other hand, new data science projects start all the time, and the transition to Julia is easy (and worth it in my opinion). That means that in my experience, Julia is mostly competing for marketshare with NumPy/SciPy/SKLearn/Pandas/R/Matlab.

[1] https://github.com/cjdoris/PythonCall.jl

Look into Julia as a promising alternative -- the language itself is superbly fast (aside from initial compilation) and there's an impressive scicomp ecosystem to say the least, all written in native Julia. This allows for program rewriting / metaprogramming more broadly and is insanely powerful once you get a feel for it.

That said, this article's story of someone using the wrong algorithm is a bad example in my view. Python hasn't succeeded because people are more likely to use more efficient algorithms due to easier experimentation, it has succeeded because the of the size of the ecosystem and the fact such algorithms are easily available.

[1] https://youtu.be/4RSht_aV7AU

[0]: https://nim-lang.org/

There are so many ways to slice an array or a dataframe, and only a few of them are valid Python code.

Keeping the language API should have been a priority, but that is a consequence of operator overloading features.

Is this about things you want to be able to express in syntax but can't? Or the other way around - stuff that uses syntax/operators but should really be methods?

i.e. some_array[0:5,0] which isn’t valid pure Python notation.

Open ipython3

    In [3]: class Test:
       ...:     def __getitem__(self, index):
       ...:         print(index)
       ...: 

    In [4]: Test()[1, 2, 1:3, ..., :]
    (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None))

Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.

You seem to have a much better grasp of Python than us, would you mind posting an example where the snipped I posted successfully accesses data from an array in pure Python? That way I can not only take the L, but correct the record and learn something in the process.

https://www.pythonmorsels.com/p/2rk5t/

Is laziness a virtue? I reused the slicing implementation in `range(foo)[index]` so I didn't need to do that logic myself.

Maybe you're thinking of things like x[:, np.newaxis] where x is a numpy array? This is valid Python code outside of numpy as well, although the built-in data structures like lists and dicts won't know what to do with the :.

Numpy and Pandas libraries have some non-standard ways to slice arrays, get the subarrays and the data from them.

The reasons for python’s success are obviously the ecosystem. Numpy is the foundation. On top we have sklearn, statsmodels, pandas, matplotlib. Before our project most work in the department was done in Stata, a proprietary language/tool that works well for some classical regression and stats work but falls apart as soon as things get complicated. Moving to python allowed us, a group of social scientists, to work on some really hard problems.

Now we have boosted tree models and other tools that just can’t be used in the old tools like Stata.

Python and R run the show in social science.

[1] Numeric age for D: Mir GLAS is faster than OpenBLAS and Eigen:

http://blog.mir.dlang.io/glas/benchmark/openblas/2016/09/23/...

[2] There are two types of data scientists — and two types of problems to solve:

https://medium.com/@jamesdensmore/there-are-two-types-of-dat...

[3] Adding ANSI C11 C compiler to D so it can import and compile C files directly:

https://news.ycombinator.com/item?id=27102584

One is wary of one-dimensional analysis of anything in a software context.

Who cares if the Fortran library runs like the blue blaze, if it cannot be readily maintained?

Displacing Fortran was a side-effect of that trend, as now it wasn't about productionizing Matlab code into Fortran, but Python could do general purpose computing adequately as well.

As the article notes, various numerical kernels have been wrapped as Python compiled modules/libraries, and numpy and other systems seem to work OK for many applications.