> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
> You already know[0] the answer to the most of that question, so I don't know why you're asking that part again.
My point is that once you start editing (either by hand or through a tool) the version numbers in the second section, then that's no longer cached state derived entirely from the go.mod files of your dependencies which can be regenerated from scratch. It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
Imagine you:
1. Add a dependency on foo, which depends on bar 1.1.
2. Decide to update the version of bar to 1.2.
3. Commit.
4. Weeks later, remove the dependency on foo and tweak some other dependencies. Tidy your go.mod file.
5. Change your mind and re-add the dependency on foo.
At this point, if you look at the diff of your go.mod file, you see that the indirect dependency on bar has changed from 1.2 to 1.1. Is that because:
A. You made a mistake and accidentally lost a deliberate upgrade to that version and you should revert that line.
B. It's a correct downgrade because your other dependency changes which have a shared dependency on bar no longer need 1.2 and it is correctly giving you the minimum version 1.1.
Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
> In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
I don't know. I guess I just don't understand the system well enough.
> It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
> Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
I'm not aware of any package management system that will remember dependencies you no longer depend on, except by human error when you forget to remove that dependency but keep punishing yourself and others by making them build and install that unused dependency. No matter where the information lived before it was deleted, it's still up to the human to do the right thing in a convoluted scenario like you're describing.
> With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
That doesn't solve anything if you don't look go back to the commit that had that information. If you do go back to that commit, you have all the information you need right there anyways. You can add your own comments to the `go.mod` file, so if something changed for an important reason, you can keep track of that (and the why) just as easily as you can in any other format. Actually, easier than some... does package.json even support comments yet? But it only matters if you go back and look at the previously-deleted dependency information instead of just blindly re-adding it as I imagine most people would do, which is a huge assumption.
>> It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
> This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
> I don't know. I guess I just don't understand the system well enough.
I've done my best to explain it to you, literally multiple hours of my day writing these comments. Maybe I suck at explaining, or maybe you're not interested in what I have to say, or maybe I'm somehow completely wrong on everything (but no one has bothered to explain how). "More or less the same" is not the same. The nuances make a huge difference, and Go Modules has done things incredibly well on the whole. Package managers aren't a zero sum game where you shuffle a deck of pros and cons, and you end up with the same number of pros and cons at the end.
> I've done my best to explain it to you, literally multiple hours of my day writing these comments.
Not the op, but wanted to express appreciation here. I read all your comments and feel like I gained a lot of clarity and insight from them. Would love to read your blog if you had one.
I appreciate the time you put into these comments and definitely learned a lot from them.
My original point was just that the article reads like an incorrect indictment of all other package managers that use lockfiles. Whether Go does things better is for most part orthogonal to what I was initially trying to get at.
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
[0]: https://news.ycombinator.com/item?id=30870862
[1]: https://github.com/caddyserver/caddy/blob/master/go.mod