There are certain ways you can objectively determine the security of a package:

- If it or one of its dependencies has been compromised

- If it and its dependencies are made by a trusted org (e.g. I would trust something from Microsoft more than something from a random developer)

- If it and its dependencies have been audited and verified (this could even be a quality audit, or something like the Elm ecosystem)

- If it and its dependencies are made by you, you can be pretty confident that nobody has tampered with them

Similarly there are ways to determine the quality (e.g. code coverage, usage, whether it has tests). And of course you can see the license.

But state these things directly, don't try to summarize them with numbers. Because ultimately those numbers are almost meaningless and make your tool look fake.

What's interesting is that arbitrary, unit-less scores seem to work well in products like Lighthouse [1] and SSL Labs [2] even though they're also "basically made up" metrics. I'm curious if anyone has ideas about why this may be the case.

To your point about just stating things directly, we also do that too. See https://socket.dev/npm/package/left-pad/issues, for example.

[1]: https://developers.google.com/web/tools/lighthouse [2]: https://www.ssllabs.com/ssltest/index.html

I love the direction you're going with this. Would love to hear more about:

1. Status. Is this a beta product or do you have enough here to sell a service?

2. Roadmap. What are you building out next? What are core capabilities you see being added to Socket?

3. Pricing. How much does it cost? What's your revenue model?

Depending on how you've put this together you may be addressing a serious need in this space that I've only previously seen addressed by stuff like Deno (where you set application capabilities prior to execution) or via virtualization.

Contact details in my profile. If your product does what it says on the tin I'd love to give it a spin.

2. https://socket.dev/roadmap See https://socket.dev/npm/issue for analysis we have developed on a per-package basis. We are working on integrating these with our GitHub App to provide custom tailored project views into this data. More info here https://socket.dev/blog/inside-node-modules

3. It will be a paid product, but is free while it is in beta. We plan on keeping it free for open source.

We are looking into providing support for the Deno ecosystem down the road as well. The capabilities stuff they have is super great, but you lose all of that benefit for every dependency as soon as you turn it off, so we think there is probably room for this kind of analysis there. Hopefully Socket can provide a similar signal that --allow-net provides, but for all of npm!

[1] https://socket.dev/npm/package/lodash/files/4.17.21/package....

[2] https://directory.fsf.org/wiki/License:Expat

If you find more examples like this, please email them to me (see bio) so we can fix them. We appreciate the feedback!

Also "left-pad Supply Chain Security: 0", lmao.

Packages can be finished and do everything they set out to do. The fact they haven't been updated in a while can be a measure of high quality as much as it can be a sign of low quality.

So is Socket.dev open source? On your Github (https://github.com/SocketDev) I cannot find the Github App source code. So being such proponents of open source, why don't you release it as open source?

As a maintainer I know how much time it takes to run an open source project beyond just writing code. There's triaging issues, reviewing PRs, answering questions, adding support for alternative operating systems/runtimes, and more. At this precise moment we just don't have that capacity, although I look forward to the day when we'll be able to share more of the code publicly.

Also, for the record, the Socket GitHub app was designed to only read package manifest files such as `package.json`, `package-lock.json`, and `yarn.lock`. It doesn't read other files, and it definitely doesn't send those files to remote servers.

There is both a large need for improvements in npm supply chain security and a market willing to pay for them.

Concerns:

1) npm Open-Source Terms condition four[1] states.

You may access and use data about the security of Packages, such as vulnerability reports, audit status reports, and supplementary security documentation, only for your own personal or internal business purposes. You may not provide others access to, copies of, or use of npm data about the security of Packages, directly or as part of other products or services.

This statement seems vague enough to potentially include your use case. It also seems to include what snyk, jfrog xray, sonatype, and white-source do, so maybe this is not an issue.

2) It appears that this will be an open-core business. What capabilities are you willing to provide in the free/community edition and under which licenses?

3) The website doesn't show pricing. Can you provide details on this?

Questions:

1) What are your thoughts on using reproducible builds[2] plus Diverse Double-Compiling (DDC)[3] on the dependency graph to ensure build artifacts originate from known git repositories? Disclosure, I've been working on this for a few months now.

2) Where do you run your analysis? AWS and DigitalOcean have terms that prevent running high risk code.

3) Do you have examples of previous attacks and how your tooling would handle them?

Best of luck.

[1] https://docs.npmjs.com/policies/open-source-terms#conditions [2] https://reproducible-builds.org/ [3] https://dwheeler.com/trusting-trust/

2 and 3) We're still figuring out the business model, but here's our current plan: Package search and Package Health Scores are free for everyone to use through our website https://socket.dev.

Socket integrations, such as the GitHub App, are free for open source repositories forever. For private repositories, Socket is free while we're in beta, but we'll eventually charge something like ~$20/developer/month for private repos. We're still working out pricing but our #1 aim is to keep it affordable so everyone can get protected.

Question 1) I love this idea! This is something the team is already talking about. We want Socket to report reproducible builds and use them as a positive signal, as well as highlight them as a badge on the package page. For npm packages, lots of them probably already have reproducible builds that we can check by just running `npm install; npm build; npm pack`. I need to think more about DDC and how that would fit it. Perhaps we can chat about it sometime?

2) We're currently doing static analysis, so not actually running the code. Our dynamic analysis isn't ready yet so we'll cross that bridge when we get there.

3) All of the issues that Socket detects were picked with previous npm supply chain attacks in mind. You can see a list packages npm removed for security reasons here: https://socket.dev/npm/category/removed When you view any of these, we show the results of our security analysis. Here is a removed package I just picked at random to give you an idea:

https://socket.dev/npm/package/netlify-swag/files/1.2.0/inde...

I became aware of his existence after I realized he was clever enough to set up a bot to watch new posts on https://web.dev (I used to be content lead for that site) and automatically post every new thing here on HN

The single biggest reason that NPM is so vulnerable to supply-chain attacks is that running `npm install` or `npm update` will always install the absolute latest semver-compatible version, regardless of how long that version has been published or any other criteria.

If suppose those two NPM commands instead had flags like --conservative, --default or --canary (OK, I suck at naming, but you get the idea) where, for example:

1. --conservative installs the most recent version has been released for at least 2 weeks or has at least 10000 downloads.

2. --default only requires the most recent version to have been released for at least 1 day or has at least 1000 downloads.

3. --canary would give you the existing behavior, where you get the most recent semver-compatible version even if it's only a second old.

The idea being that, depending on the risk level of your project, you could use a different flag. Furthermore, security researchers and companies like Snyk could install packages in --canary mode in sandboxes to look for malware.

The fact is that despite all the (deserved) handwringing about NPM supply chain attacks like with ua-parser-js and the faker/color fiasco, most of these issues are discovered very quickly. Minor changes to NPM would provide a huge amount of additional security (for those who need it) with extremely little cost.

Initial impressions when visiting the website though is I can't understand how it works exactly, for example the learn more page at https://socket.dev/integrations says Socket helps improve the posture, but it would be good to explain (perhaps with screenshots an actual example?) of it catching an issue.

Right now the integration is fairly slim, it just does typo squat warnings when you install a package that has a similar name to a more common package. The warning comes in the form of a comment in PRs that include additions of packages that meet this criteria.

We have a bunch of other detections listed here: https://socket.dev/npm/issue which have not been added to the GitHub App yet, but are available on a per-package basis for manual research at the moment. Over the next few release cycles we will be adding additional issue checks and warnings to the Github integration so that you can get a warning when dependencies add new capabilities, add suspicious things like analytics or install scripts or add unknown publishers to their maintainer list, or start publishing binary or obfuscated code. These will be automatically turned on and rolled out as we determine them to be not too noisy and provide interesting signals.

Sorry about the confusing page. We're still working on some of the pages on the site. You might find these links more informative:

- Launch post: https://socket.dev/blog/introducing-socket

- What's Really Going On Inside Your node_modules Folder? https://socket.dev/blog/inside-node-modules

- And maybe even the launch Twitter thread: https://twitter.com/feross/status/1498676284590800903

https://socket.dev/npm/issue/unclearLicense

You might enjoy this blog post which explains supply chain attack at length and starts off with a concrete example of a recent supply chain attack (the ua-parser-js attack): https://socket.dev/blog/inside-node-modules