I'm just dumbfounded that HN guys became so negative! Literally every comment is so negative :( Please guys, this guy did an awesome job, the painful slowness of npm -- I gotta endure it everyday, this guy is literally giving me a sigh of relief. I'm gonna use it over npm and even contribute if necessary.
If "written in Rust" is a clickbait, then most first page HN titles are clickbaity too otherwise why will you click on them? To me and most people, "Written in Rust" or even "Written in Go" reads like "it's fast" and that's definitely a feature!
HN is extremely JavaScript-fatigued. Feels like all JavaScript-related novelty has been getting downvoted lately unless it's coming from a small set of "trusted" JS authorities (Next.js maintainers, Kent Dodds, etc.)
My comment was praising this person, and I got downvoted for it.
They've contributed so much work, and I think it's really cool that they're always diving in to solve problems and take a crack at alternative solutions. Mad props.
I kind of disagree, given that I wrote a comment that perhaps questioned the project somewhat. A lot of comments are pretty positive, at least of the time of this writing. I started out with "cool project". Even if I use node every day like yourself the startup time for npm commands is so fast compared to other stuff I use like transpiling typescript to javascript and the worst offender of all: Babel.
The babel transpiling takes about 5 seconds compared to the 200ms of npm scripts startup time. Since I haven't felt this pain of npm startup times myself as a pain I must be able to state that without being interpreted as negative, don't you agree?
Of course, speed increases are positive over all but this requires me to install something just to gain 200ms from whenever I run an npm command which isn't that often. I am sure there's a lot of people out there who this benefits a lot but I haven't personally experienced this is my years of doing node development.
There have been performance-focused alternatives to NPM for some time: Yarn and PNPM. They offer other benefits as well, and both have become pretty popular (Yarn moreso I believe). There are performance-focused alternatives for a lot of things in the Node/JS ecosystem. TypeScript tsc: ESBuild. Babel: SWC. NVM: Volta. All popular to varying degrees, with people happy to install them for their performance gains.
Sure, npm run isn’t my biggest bottleneck. But it’s noticeable, and becomes annoying a couple of layers of indirection deep… which isn’t that uncommon, especially in monorepos. Shaving 200ms off might not sound like much, but it really depends how often you’re taking that 200ms hit.
It's a good direction, like swc which replaces babel, webpack and esbuild, and is also written in rust. I feel all these parts needs to be done in a language like rust, not just some parts. I wish the whole "npm install" thingy aka the package manager which is painfully slow, needs to be written in Rust next. That would be awesome!
Agree on the negativity (surprising and completely unwarranted), but a nitpick: npm install is slow (this doesn't replace that). NPM scripts are generally very fast, unless the implementation of your actual script is slow (in which case it will be similarly slow with Dum).
(the Dum GH has benchmarks showing perf. gains for scripts, but the % improvement is only significant because the script doesn't do anything - the gains would be static for real-world use-cases, making the % gain negligible)
All that said, there are other very valid reasons to use Dum - limited, and containing a lot of caveats, but very valid all the same.
> Instead of waiting 200ms for your npm client to start, it will start immediately.
That seems like a pretty big performance problem on Node's part.
Wouldn't it be better if Node solved this, than people having to use external package managers that may not be so well-integrated into the ecosystem? Thirdly would people even bother using an external package manager because of a 200ms delay? Is that considered acceptable at this point?
Just spent a few minutes checking NPM's startup code. For a total run of ~234ms, it spends:
~50ms loading a version-parsing lib, loading JSON metadata containing Node binary version and checking node binaries against some hard-coded compatibility data. Dum doesn't seem to do any such sanity checks, which may be fine if you manually ensure your environment is correctly setup.
~25ms initiating logging (& exception handling). This seems like it could be improved.
~40ms loading config & 2 external dependencies (`which` and `graceful-fs`) - not sure if the config is the reason for slowness. The dependencies bring functionality Dum doesn't seem to need as much as NPM does. I guess the load-perf of the dependencies could be improved, unless it's the config loading causing this.
~80ms checking for updates!!! Absolutely no need to do this on every run - this could be throttled somehow at the very least.
The remaining ~40ms is actually running the script
---
Update: After looking more closely, it appears the update checking code doesn't actually run every time - it's taking 80ms just to load that code (without doing the actual check - the check is cached)
It's not solely a node problem. The benchmarked task is starting up a node subprocess, that's what most of the 50ms in the Rust execution is. So npm is adding an additional 150ms + whatever the Rust code takes, beyond shelling out to start a node subprocess.
It would be better, but it might also be borderline impossible or at least extremely difficult. Since Node is open source, anyone is free to try to solve. Maybe it is trivial!
In the meantime this is a practical, acceptable solution.
> Thirdly would people even bother using an external package manager
What do you mean? This tool is not a package manager, it is just a script runner.
Interesting, I had seen previous work on this and thought they’d already squeezed the available perf juice out of it. Nice to see there’s still more room for improvement on this front. And especially exposing more to userland, I definitely have use cases for that!
> It would be better, but it might also be borderline impossible or at least extremely difficult.
Out of curiosity, what makes you say that?
> What do you mean? This tool is not a package manager, it is just a script runner.
Bah, sorry. I had PNPM in mind when I wrote that (which I highly recommend). PNPM doesn't explicitly focus on startup time, but it does majorly improve performance as it caches packages.
node has a costly and intertwined init itself, where the startup cost is 50-100ms.
On top of that, `npm` similarly has a high startup cost adding to the original node cost.
Node apps are very slow to start to run due to synchronous `require` eager-loading all the module files, so there is some re-architecting necessary to get a performant startup time
yarn v1 is probably the best of the bunch with startup lag close to pure node (100-150ms)
yarn v2 and up can be extremely bad, 1s-3s with medium to large repos, especially with PnP enabled which AFAIK forces eager resolution.
pnpm similarly has had troubles starting up quickly. It used to be even slower as it passed the command to `npm` in a suboptimal way. Nowadays its about the same speed as `npm` as it uses `npm` for `run`.
Honest question, what use case have people run into with non scripts that a 300ms startup time is painful for? Mine have always been simple and run not very often, so it’s barely mattered to me, but I’m sure I’m missing some obvious uses that others have
That's a question designed to confirm priors, depending on perspective it's not necessarily an honest one. It's better to think about changes like this in terms of what becomes possible with overheads removed ("what becomes more convenient to do as an npm script if startup is much faster?") or comparable examples from other areas ("what happened to the web when it no longer became necessary to wait 300ms+ for application logic to respond to a click?") etc.
Sure, that’s what I was trying to get at but worded badly. What am I missing that this would help with, and that NPM scripts are the right choice for them?
Git hooks as someone mentioned below are a good one.
It absolutely was an honest question, and nitpicking on my intent to imply I think this project is silly seems a little unnecessary: I don’t, and really do want to know what it could help with, as none of my use cases need it right now. I’d like to keep any eye out for when they would though.
For me, speedup enables a lot of much simpler ways to do the stuff that I already do.
Run tests and linter on-save. Have vim with LSP and other tooling, startup fast enough to close and re-open the entire editor, rather than background (<CTRL>-z) or alt-tab. Just do a <CTRL>-c<CTRL-p><enter> to reload any changes in a running dev server, rather than tooling that attempts to listen to io-events and reload the code magically.
There are loads of small things that become a lot simpler when kill&start is just as fast as "internal reload".
I've always used --watch just so my stuff would re-run without me having to manually do it...? That watch modes were to get around slow startup, and not a usability feature, is new information to me.
Ah that’s fair. Most of the tooling I’m running is still honestly pretty slow (CRA being the main set, Jest being the other), and they both do the “watch” trick.
Anything that takes 300ms to start up is not suitable for anything but server applications (and even then...) or daemons. Nothing living in userspace should be that slow, IMO.
Maybe for stuff that runs over and over again and needs every microsecond it can get. But if that's a requirement, nodejs seems like a pretty bad choice anyway.
It adds up. 300ms is pretty good, but that's just starting up a node process, then you get a long process of parsing and executing the script itself.
Which initially is not too bad, but the bigger the codebase gets and the more often you run it, it adds up. 300ms per invocation is something that a lot of CI providers would love to save.
That doesn't apply here: all these scripts do is shell out to some other binary. Any performance cost of using node or other binary is going to be exactly the same. It's 200-300ms regardless of what the underlying script does.
Especially when you have 6 git hooks, and they're all run one by one in "pre-commit", which is written in Python, another whole slow interpreted ecosystem. I do not miss my last job.
You could make this actually useful by making it read a .env file to set the environment, instead of what people always do, which is:
> "test": "NODE_ENV=test mocha --reporter spec"
(and doesn't work across platforms like windows)
Or added functionality like being able to pipe the output to multiple places (eg. terminal and log file).
That would offer perhaps a slightly more compelling use-case than saving 200ms; I don't care about that much, I honestly doubt anyone else does either.
The spawn time for an npm compared to the runtime for an npm task are orders of magnitude different, surely, for any non trivial task.
"NODE_ENV=test mocha --reporter spec" actually works on Windows when using recent versions of Yarn. They implemented a minimal shell parser for this and similarly basic features.
I'm glad Yarn supports this natively, though. It seems silly to rely on an external package for something so basic (especially with the security vulnerabilities of the ecosystem as a whole).
> NOTICE: cross-env still works well, but is in maintenance mode.
Which is why I recently kicked all the cross-env out of my nodejs projects and replaced it with Ruby's dotenv[0] (in projects that already used ruby alongside) and zenv[1], in Rust for projects that don't.
My first thought seeing this was “I wonder if they know about bun”, was happy to see it’s an inspiration! And just in general happy to see more performance-focused tooling projects in the ecosystem. Good stuff.
Rust based tools are so cool. I have used various cli tools and they are just awesome. One example is fnm which is fast and makes it easy to switch node js versions.
Considering all the "--watch" and "watch" options and scripts that come with many default tools (jest, react-boilerplate etc), it is considered a problem to many.
For me it is a serious nag. And I come from Ruby, where, with Rails, startup time is abysmal. So I am used to something.
Considering all the "--watch" and "watch" options and scripts that come with many default tools (jest, react-boilerplate etc), it is considered a problem to many.
Don't people use the watch options so they don't need to interact with the app to have it rerun things? I don't think speeding up how long scripts take to start will change that.
In my case, it hooks into my editor or IDE. E.g. I don't need to have a "--watch" feature in my test-suite, because my Editor/IDE can fire the tests just fine.
Same for (re-)starting webservices and such. And when not available or applicable in my editor, GNUtools, bash and unix offer plenty of tools to easily do this for me. E.g. https://linux.die.net/man/1/inotifywait to fire a command (e.g. restart a service) on changing files.
I presumed these `--watch` where there to solve the slow-booting issues only (like in Rails) but maybe they are there because someone didn't know their OS and DE could do it for them already and decided to build it into the suite or tool instead of using existing tooling?
i've been loving egoist/dum as a complement to antfu/ni, but god forbid i ever remember how to update a cargo package. egoist if you're here, have you considered publishing binaries to npm?
This is one of those people who just constantly pump out so much open source code. I remember looking through their profile when I was using EME years ago (2017?), wishing I had the time to do half of what they were tackling.
Slightly off topic: So many projects and posts boast “written in Rust”. I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?
From a quick glance, it looks like this could have been implemented in any native language. I like Rust as much as the next guy, but using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.
I've seen myriad "written in C/Python/Java/JavaScript/TypeScript/Lua" posts over the years. You're not imagining things, though. The world waited 30 years (from the 1985 when C++ appeared to 2015 when Rust 1.0 was released) for a new, viable language in the systems space (read: without garbage collection). That's a really long time, and the excitement around the language is well deserved.
Of course I thought of go – it's just not in the systems language space (it has GC). (I believe Rob Pike expressed some remorse for causing some confusion about that by using that phrase in early go documentation. I remember seeing him on stage on a 4-member panel expressing regret, anyway). So it really had been 3 decades.
>I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?
There are a few projects that tout being written in C++, but it has never been a regular "selling point" in HN, because HN was a thing almost 2 decades after C++ was the "new hotness".
So, we've seen it for Node, Rust, and Go more.
>I like Rust as much as the next guy, but using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.
You think this person spend weeks writing this for "clicks"? Unless the author posted it, they might not even know it was posted on HN (it's often the case with posted projects: someone else finds them and posts them).
They had a pet project in mind, they liked Rust and wanted to use it, they did.
> was there a point in time where “written in C++” was a similar “selling point”?
no
> using it to get clicks when Rust specifically over another native language doesn’t make a difference feels a bit clickbaity.
- every single article mentioning rust in any way has this complaint in the comments
- I think that much of the reason people mention it is not a cynical ploy for clicks (though that exists for sure), but denotes that the program is a modern program that is likely to work in a certain way? It's hard to specify what that way is, I should try and write something about it, but I think people who complain about this are missing that it's kind of a cultural signifier more than a literal statement about the language it's written in.
It's not a ploy for clicks. Programmers spend a lot of time working with programming languages and so the choice of language is an interesting thing to mention, no matter what it might be.
Personally I rather like that convention (and similarly for Go). Often I will follow links that specify one of these two languages since I primarily work in Go and Rust, and it's useful to add them to my mental map (or favourites folder) of what is available such that if I need something similar I can find it again.
Libraries or tools written in C++ interest me less personally, because they're more work for me to integrate into whatever I'm typically doing, but I can see that someone working in C++ might like such things to be labelled the same way for the same end.
It's done for other languages, as others here have mentioned, but it happens more for Rust lately. A project being advertised as being written in JS or Java or C# does have a signal about how easy it is for the reader to contribute (if they know the language), to integrate it with other projects they use (network effects) or what their operation might be (can I build a single binary and deploy everywhere? what is the observability and debugging tooling like?).
Rust answers some of those implicit questions in a way that is acceptable to a lot of us.
There was probably a time when C++ had the kind of enthusiasm that Rust does now, but as C++ was released in ~1984, I don't know that you can quite compare the two.
Written In Rust is great because Rust is one of those Fun And Shiny languages that people want to write in just to write in it, like Elixir or Node (topical!), except it's not going to pull an Electron and involve a giant VM and a bundle system that includes a user-mode Xbox controller driver by default, but rather is faster in its first draft than you could possibly squeeze out of one of those other languages. Rust captures the 'systems they don't teach you about in college' vibe without products written in it slightly decreasing the usability of the developer ecosystem.
There was a time where anything with JavaScript in the title automatically got into the front page. And now it's Rust. Rust is now in it's golden age. The question is if all the enterprise "Java" will switch over from JS to Rust, and demand that Rust change it's syntax and methodologies so that the "JS" developers can understand it.
> Slightly off topic: So many projects and posts boast “written in Rust”. I haven’t been on HN that long—was there a point in time where “written in C++” was a similar “selling point”?
"Written in Rust" is the new "Sent from my iPhone".
You are incorrect, though. The reason people advertise tools as being written in Rust is because having tools written in Rust tend to scratch a few itches that a lot of people have: speed, the fact it is native, the safety due to the borrow checker, the ease of installation, the ease of contribution, the possibility to learn something from the code, among other things. Those things matter a lot to me, and I assume to a lot of other people.
I am probably going to look into it, since I'm often annoyed by npm run as well. However if it were in another language I don't want my tools written in, I would probably pass. Something being in a certain language is not a guarantee that it will be great, but being in others is often a guarantee that it isn't. Which language tools are written matters a lot to me.
I understand how all this might look like to someone "out of the loop", however, but I wouldn't be too quick to judge other people.
Dismissing a project out of hand because it's written in a certain language because it's trendy is the same faulty logic than choosing a certain language for a project because it's trendy.
I find I am more likely to choose it because you don't have to be super experienced in Rust to produce fast, reliable binaries. That is not the case with C/C++.
> you don't have to be super experienced in Rust to produce fast
That's an interesting take. As an old fart, I find Rust's choices new and unfamiliar. I like them, it seems very sound, but the learning curve for people used to C is real.
Isn't the point of `npm run`/`npx` to run stuff in the local `node_modules` folder?
It's much more convenient to type `npx something` than `./node_modules/something/bin/something`, for example. Especially so when folder names are not obvious.
Just put your scripts in bash functions and then all of them in a scripts file, with enhanced $PATH at the top, or use make. In fact most of the more sophisticated projects use this approach.
package.json is such a sloppy and arbitrary place for your scripts. I don't know how the creator of node looks at the mountain of things that are all in package.json and thinks that's a fine design. You are a package manager, stick to it.
I've seen an occasional `npm run bash ...` and `npm run docker ...`. I presumed that was to have tooling like .env hook into it and setup the env. But it might be stupid copy-paste errors, as well.
If "written in Rust" is a clickbait, then most first page HN titles are clickbaity too otherwise why will you click on them? To me and most people, "Written in Rust" or even "Written in Go" reads like "it's fast" and that's definitely a feature!