IMO the biggest drawback to a monolith, maybe beyond those listed, is losing the 1-1 mapping of changes to CI to releases. If you know "this thing is broken", the commit log is a fantastic resource to figure out what changed which may have broke it. You submit a PR, and CI has to run; getting most CI platforms cleanly configured to, say, "only run the user-service tests if only the user-service changes" isn't straightforward. I understand there are some tools (Bazel?) which can do it, but the vast, vast majority of systems won't near-out-of-the-box, and will require messy pipelines and shell scripts to get rolling.
There's also challenges with local dev tooling. Many VSCode language extensions, for example, won't operate at their best if the "language signaler" file (e.g. package.json for JS) isn't in the root of the repo; from just refusing to function, to intellisensing code in another project into this one, all manner of oddities.
Meanwhile; I don't think the purported advantages are all that interesting. Being able to write the blog post and make the change in the same PR? I've never been in a role where we'd actually want that; we want the change running silently in prod, publish the blog post, flip a feature flag or A/B test proportions. Even an argument like "you can add schemas and the API changes in one go"; you can do that without a "monorepo", just co-locate your schemas with the API itself, this isn't controversial or weird, this is how everywhere I've worked operates.
None of that is to even approach the scale challenges that come with monorepos at even reasonable sizes; which you can hit pretty quickly if you're, say, vendoring dependencies. Trying to debug something while the Github or Bitbucket UI is falling over because the repo is too large, or the file list is too long, isn't fun.
I'm not going to assert this is a hill I would die on, but I'm pretty staunchly in the "one service, for one deployment artifact, for one CI pipeline, for one repo" camp.
As a big fan of the monorepo approach personally, I would say the biggest benefit is being able to always know exactly what state the system was in when a problem occurred.
I've worked in large polyrepo environments. By the time you get big enough that you have internal libraries that depend on other internal libraries, debugging becomes too much like solving a murder mystery. In particular, on more than one occasion I had a stacktrace that was impossible with the code that should have been running. A properly-configured monorepo basically makes that problem disappear.
This is more of a problem the bigger you are, however.
I think we're just reducing down to "programming at scale" is hard, at some point.
Sure; that is a really big problem, and it becomes a bigger problem the bigger you are. But, as you become bigger: the monorepo is constantly changing. Google has an entire team dedicated to the infrastructure of running their monorepo. Answering the question: "For this service, in this monorepo, what is the set of recent changes" actually isn't straightforward without additional tooling. Asking "what PRs of the hundred open PRs am I responsible for reviewing" isn't straightforward without, again, additional tooling. Making the CI fast is hard, without additional tooling. Determining bounded contexts between individuals and teams is hard, without additional tooling.
The biggest reason why I am anti-monorepo is mostly that: advocates will stress "all of this is possible (with additional tooling)", but all of this is possible, today, just by using different repos. And I still haven't heard a convincing argument for what benefits monorepos carry.
Maybe you could argue "you know exactly what state the system was in when something happened", sure. But when you start getting CI pipelines that take 60 minutes to run, or failed deployments, or whathaveyou; even that isn't straightforward.
And I also question the value of that; sure you have a single view of state, but problems don't generally happen "point in time"; they happen, and then continue to happen. So if we start an investigation by saying "this shit started at 16:30 UTC", the question we first want to have answered is "what changed at 16:30 UTC". Having a unified commit log is great, but realistically: a unified CI deploy log is far more valuable, and every CI provider under the sun just Does That. It doesn't mean squat that some change was merged to master at 16:29 if it didn't hit prod until 16:42; the problem started at 16:30; the unified commit log is just adding noise.
You had the wrong tools. It doesn't matter if you have a monorepo or not, you will need tools to manage your project.
I'm on a mutirepo project and we can't have that problem because we have careful versioning of what goes together. Sure many combinations are legal/possible, but we control/log exactly what is in use.
> Sure many combinations are legal/possible, but we control/log exactly what is in use.
I'll acknowledge our tooling could have been better, but isn't it better to just be able to check out one revision of one repo and have confidence that you're looking at the code that was running?
If I have a services based architecture then I can jump straight to the repo for that particular service and have confidence that it is the code that is running.
So instead of adopting a system that makes the problem we’re discussing not possible you use a human-backed matrix of known compatible versions?
Like you do you but I’ve never seen “just apply discipline” or “just be careful” ever work. You either make something impossible, with tooling or otherwise, or it will happen.
No, it is a tool backed matrix. Illegal combinations are not possible, and we have logs of exactly what was installed so we can check that revision out anytime
To solve this properly you need to store the deployed/executed commit id of any service. That could be in the logs, in the a label/annotation of a kubernetes object or somewhere else. But this has nothing to do whether you use a monorepo or multiple smaller repositories. In some projects of me, we use the commit of the source repo as docker tag. And we make sure that the docker image build is as reproducible as possible. I.e. we don't always build with the latest commit of an internal library, but with the one that is mentioned in the dependency manifest of our build tool. Since updating all those internal dependencies is a hassle, that is updated automatically. It means there is an auto-generated merge requests to update a dependency for every downstream project. Therefore all the downstream pipelines can run their test suites before an update gets merged. Once in a while that fails, then a human has to adapt the downstream project to its latest dependencies. In a monorepo that work has to be done as well. But for all downstream projects at once.
Submodules are hell. I work somewhere with a polyrepo topology, with the inevitable "shared" bits ending up integrated into other repos as submodules. Nothing has been more destructive to productivity and caused so many problems.
It can now, but that's not the default. The defaults for submodule suck, because they match the behavior of old versions of git for backwards compatibility.
Yeah. Leaving the UX-issues aside. Don't ever use submodules to manage dependencies inside of each polyrepo, it will eventually accumulate duplicate, conflicting and out of date sub-dependencies. Package managers exist for a reason. The only correct way to use submodules is a root-level integration-repository, being the only repo that is allowed to have submodules.
The only problem I have with a monorepo, is that sometimes I need to share code between completely different teams. For example, I could have a repo that contains a bunch of protobuf definitions so that every team can consume them in their projects. It would be absurd to shove all of those unrelated projects into one heaping monorepo.
Well that's what a monorepo is! I work on one, it's very large, other teams can consume partial artifacts from it (because we have a release system that releases parts of the repo to different locations) but if they want to change anything, then yeah they have to PR against the giant monorepo. And that's good!
Teams out of the repo have to be careful about which version they pull and when they update etc. However, if you are a team IN the monorepo, you know that (provided you have enough tests) breaking changes to your upstream dependencies will make your tests fail which will block the PR of the upstream person making the changes. This forces upstream teams to engage (either by commits or by discussions) with their clients before completing changes, and it means that downstream teams are not constantly racing to apply upgrades or avoiding upgrades altogether.
I work on shared library code and the monorepo is really crucial to keeping us honest. I may think for example that some API is bad and I want to change it. With the monorepo I can immediately see the impact of a change and then decide whether it's actually needed, based on how many places downstream would break.
Ok. I've had some time to think about this, and I am warming up to the idea. It would sure simplify a lot of challenging coordination problems. My only real concern is that the repo may grow so large it becomes very slow. Doubly so if someone commits some large binaries.
I am certainly not a heavy user, but for work I've made myself a "workflow" repository which pulls together all the repositories related to one task. This works super well. There sure is a bit of weirdness in managing them, but I found it manageable. But I'll admit that I don't really use the submodules for much more than initial cloning, maybe I'd experience more problems if I did.
Yes, but it's because submodules are a badly architected, badly implemented train wreck.
There are many good and easy solutions to this problem, all of which were not implemented by git.
git is a clean and elegant system overall, with submodules as by far the biggest wart in that architecture. They should be doused with gasoline and burned to the ground.
I like using submodules for internal dependencies I might modify as part of an issue. I like conan or cargo for things I never will. I don't particularly like conan. Perhaps bazel, hunter, meson or vcpkg are all better.
But this is a discussion of dependencies between services. You need more tooling for managing inter-service dependencies as opposed to package dependencies within one monolith.
> I've worked in large polyrepo environments. By the time you get big enough that you have internal libraries that depend on other internal libraries, debugging becomes too much like solving a murder mystery. In particular, on more than one occasion I had a stacktrace that was impossible with the code that should have been running. A properly-configured monorepo basically makes that problem disappear.
On the contrary, a monorepo makes it impossible because you can't ever check out what was actually deployed. If what was running at the time was two different versions of the same internal library in service A and service B, that sucks but if you have separate checkouts for service A and service B then it sucks less than if you're trying to look at two different versions of parts of the same monorepo at the same time.
There is no source of truth for "what was deployed at time T" except the orchestration system responsible for the deployment environment. There is no relationship between source code revision and deployed artifacts.
Hopefully you have a tag in your VCS for each released/deployed version. (The fact that tags are repository-global is another argument for aligning your repository boundaries with the scope of what you deploy).
Why not? I’m doing it right now. The infrastructure is versioned just like the app and I can say with certainty that we are on app version X and infra version Y.
I even have a nice little db/graph of what versions were in service at what times so I can give you timestamp -> all app and infra versions for the last several years.
Unless your infrastructure is a single deployable artifact, its "version" is a function of all of the versions of all of the running services. You can define a version that establishes specific versions of each service, but that's an intent, not a fact -- it doesn't mean that's what's actually running.
Am I missing some nuance here? Yes the infra version is an amalgamation of the fixed versions of all the underlying services. Once the deploy goes green I know exactly what’s running down to the exact commit hashes everywhere. And during the deploy I know that depending on the service it’s either version n-1 or n.
The kinds of failures you’re describing are throw away all assumptions and assume that everything from terraform to the compiler could be broken which is too paranoid to be practically useful and actionable.
If deploy fails I assume that new state is undefined and throw it away, having never switched over to it. If deploy passes then I now have the next known good state.
Oh, this implies you're deploying your entire infrastructure, from provisioned resources up to application services, with a single Terraform command, and managed by a single state file. That's fine and works up to a certain scale. It's not the context I thought we were working in. Normally multi-service architectures are used in order to allow services to be deployed independently and without this form of central locking.
If what was deployed was foo version x and bar version y, it's a lot easier to debug by checking out tag x in the foo repo and tag y in the bar repo than achieving the same thing in a monorepo.
I'm not sure I understand how that scenario would arise with a monorepo. The whole point of a monorepo is that everything changes together, so if you have a shared internal library, every service should be using the same version of that library at all times.
And every service deploys instantly whenever anything changes?
(I actually use that as my rule of thumb for where repository splits should happen: things that are deployed together should go in the same repo, things that deploy on different cycles should go in different repos)
Not necessarily instantly, but our CD is fast enough that changes are in production 5-10 minutes after hitting master.
But what's more valuable is that our artifacts are tagged with the commit hash that produced them, which is then emitted with every log event, so you can go straight from a log event to a checked-out copy of every relevant bit of code for that service.
Admittedly this doesn't totally guarantee you won't ever have to worry about multiple monorepo revisions when you're debugging an interaction between services, but I haven't found this to come up very much in practise.
Edit: I should also clarify, a change to any internal library in our monorepo will cause all services that consume that library to be redeployed.
> What do you do with libraries shared between different deployment targets?
The short answer is "make an awkward compromise". If it's a library that mostly belongs to A but is used by B then it can live in A (but this means you might sometimes have to release A with changes just for the sake of B); if it's a genuinely shared library that might be changed for the sake of A or B then I generally put it in a third repo of its own, meaning you have a two-step release process. The way to mitigate the pain of that is to make sure the library can be tested on its own without needing A or B; all I can suggest about the case where you have a library that's shared between two independent components A and B but tightly coupled to them both such that it can't really be tested on its own is to try to avoid it.
That's a great test and I think an argument for monorepo for most companies. Unless you work on products that are hermetically sealed from each other, there's very likely going to be tight dependencies between them. Your various frontends and backends are going to want to share data models for the stuff they're exchanging between them for example. You don't really want multiple versions of this to exist across your deployments, at least not long term
I think it's maybe an argument for a single repo per (two-pizza) team. Beyond that, you really don't want your components to be that tightly coupled together (e.g. you need each team to be able to control their own release cycles independently of each other). Conway's law works both ways.
If they have independent release cycles, they shouldn't be tightly coupled (sharing models etc. beyond a specific, narrowly-scoped, and carefully versioned API layer), and in that case there is little benefit and nontrivial cost to having them be in a monorepo.
Not GP, but I use versioned packages (npm, nuget, etc) for that. They're published just like they're an open source project, ideally using semantic versioning or matching the version of a parent project (in cases where eg we produce a client library from the same repo as the main service).
I've had the exact opposite experience. We have a polyrepo setup with four repos in the main stack (and a comical number of repos across the entire product, but that's a different story). My top pain point - almost painful enough to force a full consolidation - is trying to find the source of a regression.
When semi-regularly discover regressions on production and want to know when it was introduced. Any other project I've worked on, that can be done with a simple git bisect. I can tell you that trying to bisect across four repos is not fun. If everything were in a monorepo, I would be able to run the full stack at any point in time.
Now, if all your APIs are stable, this won't be as bad. But if you're actively developing your project and your APIs are private, I can only assume this pain will be ever present.
I think my counterpoint is: Generally, if I'm playing the part of the owner of some system N layers deep in the rats nest of corporate systems; I don't even want to think specifically about what broke. I know the dependencies of my system; if I have a dependency on the Users Service, and it looks like something related to the Users Service broke, my first action is probably to go into their slack channel and say "hey, we're seeing some weird behavior from the Users system; did y'all change something?"
At the end of the day; they're going to know best. Maybe code changed. Maybe someone kubectl edit'ed something manually. Not everything is represented in code.
The problem is that in microservice environments, a lot of complexity and source of bugs are (hidden) in the complex interactions between different components.
I also believe that this mentality of siloing/compartmentalization and habit of throwing things over the fence leads to ineffective organization.
After close to a decade of working in various microservice based organizations, I came to a big-ish monolith project (~100 devs). Analyzing bugs is now fun, being able to just step through the code for the whole business transaction serially in a debugger is an underrated boost. I still need to consult the code owners of a given module sometimes, but the amount of information I'm able to extract on my own is much higher than in microservice deployments. As a result, I'm much more autonomous too.
> Maybe code changed. Maybe someone kubectl edit'ed something manually. Not everything is represented in code.
That's honestly one of the big problems in microservices as well.
> After close to a decade of working in various microservice based organizations, I came to a big-ish monolith project (~100 devs). Analyzing bugs is now fun, being able to just step through the code for the whole business transaction serially in a debugger is an underrated boost. I still need to consult the code owners of a given module sometimes, but the amount of information I'm able to extract on my own is much higher than in microservice deployments. As a result, I'm much more autonomous too.
Could you expand how do you manage ownership of this monolith? Do you run all the modules in the same fleet of machines or dedicated? Single global DB or dedicated DB per module (where it makes sense, obviously)?
Because where I work we have a big monolith with a similar team size and it's a royal PITA, especially when something explodes or it is going to explode (but we have a single shared DB approach, due to older Rails limitation, and we have older Rails because it is difficult to even staff a dedicated team that take care of tending the lower level or common stuff in the monolith).
> Could you expand how do you manage ownership of this monolith?
We have a few devops teams (code + deployment) and platform teams (platform/framework code), the remaining teams (which form the majority of devs) own various feature slices. The ownership is relatively fluid, and it's common that teams will help out in areas outside of their expertise.
> Do you run all the modules in the same fleet of machines or dedicated?
Not sure if I understand. All modules run in the same JVM process running on ~50 instances. There are some specialized instances for e.g. batch processing, but they are running the same monolith, just configured differently.
> Single global DB or dedicated DB per module (where it makes sense, obviously)?
There is one main schema + several smaller ones for specific modules. Most modules use the main schema, though. Note that "module" here is a very vague term. It's a Java application which doesn't really have support for full modules (neither packages nor Java 9 modules count). "module" is more like a group of functionality.
> and we have older Rails because it is difficult to even staff a dedicated team that take care of tending the lower level or common stuff in the monolith).
This is usually a management problem that they don't pay attention to technical debt and just let it grow out of control to the point where it's very difficult to tackle it.
The critical part of the success of this project is that engineering has (and historically had) a strong say in the direction of the project.
But aren't micro-service specifically designed to be able to split responsibility of a large system between multiple teams. If everybody debugs and fixes bugs across the whole landscape, than everybody has to be familiar with everything, which means you are loosing the benefits. Occasionally, it might be helpful to debug the whole stack at once. But I wouldn't trust a landscape where that is needed too often. I might be that the chosen abstractions don't fit well.
> But aren't micro-service specifically designed to be able to split responsibility of a large system between multiple teams.
That's the idea, but business transactions usually span multiple services and bugs often aren't scoped to a specific service.
> If everybody debugs and fixes bugs across the whole landscape, than everybody has to be familiar with everything
A lot of things can be picked up along the way while you're debugging, and I'm usually able to identify the problem and sometimes even fix it.
> I might be that the chosen abstractions don't fit well.
Very often the case. Once created, services remain somewhat static, their purpose and responsibility often gets muddy. Mostly because "refactoring" microservice architecture is just very expensive and work intensive. Moving code between modules within a monolith is rather easy (with IDE's support), moving code between services is usually not trivial at all.
But that's just that one scenario you've described.
It's also common that if you have a dozen repos that maybe only one has changed and so when there is a defect it's trivial to determine what caused the regression.
I don't think mono or poly repos are better when it comes to triaging faults. They each have strengths and weaknesses.
>> I'm pretty staunchly in the "one service, for one deployment artifact, for one CI pipeline, for one repo" camp
This seems reasonable if nothing is shared. If there are any shared libraries then you are back to binary sharing (package managers, etc.) with this approach.
This looks trivial now, but when you multiply the number of directories by 8 or so it becomes a very nasty mess very quickly.
I think that the idea of only running what changed makes a lot of sense, I just think that managing that in declarative yml falls apart VERY quickly once you hit an inkling of scale.
I just want to comment that you are correct. Bazel allows that and so should any tool that can build dependencies DAGs. Once you have that it's absolutely feasible.
The major issue is that you need to be diligent at bookkeeping your dependencies. Bazel enforces that in the BUILD files and since everything is run in a sandbox you can't easily take shortcuts or you'll get missing dependencies errors.
>Many VSCode language extensions, for example, won't operate at their best if the "language signaler" file (e.g. package.json for JS) isn't in the root of the repo; from just refusing to function
With VSCode, a workaround is to use workspaces. Define a workspace and add each subproject folder as its own entity. VSCode will treat each folder as a project root where the language specific tooling will work as expected.
Your examples of CI and VSCode are on point, and in the bigger picture it's always about tooling.
The mono/multi repo argument is fundamentally boring to me because it always boils down to whether the shape of the tooling problem is easier to work with on this or that side of the divide.
The answer is always whichever tradeoffs work best for your situation, and the reason at the end of your post is as good of a reason as as any.
> If you know "this thing is broken", the commit log is a fantastic resource to figure out what changed which may have broke it. You submit a PR, and CI has to run; getting most CI platforms cleanly configured to, say, "only run the user-service tests if only the user-service changes" isn't straightforward. I understand there are some tools (Bazel?) which can do it, but the vast, vast majority of systems won't near-out-of-the-box, and will require messy pipelines and shell scripts to get rolling.
I'm not very familiar with the recently trending monorepo tools, but don't they generally provide a way to declare the dependencies between subrepos and prevent each subrepo from importing or otherwise depending on anything outside of those declared dependencies? If that's the case, then wouldn't CI be able to use that same dependency graph to determine when it needs to rebuild/redeploy each particular subrepo?
Well, there aren't "subrepos" … it's all one giant monorepo.
And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's typically not recorded. (Excepting bazel, which set out to solve this problem; lo and behold it was designed by a company with a monorepo, too.)
Some CI systems I've seen have half-assed attempts at it, such as "only run this CI step if the files given by this glob changed". But a.) it requires listing a transitive list of all globs that would apply to the current step, so it's not a good way to manage things and b.) every time I've seen this mis-feature, "change" is described as "in this commit"; that's incorrect. (I have base commit B, I push changes '1 and '2, for commit graph B - '1 - '2 ; CI detects for a step that the globbed files didn't change in '2, and ignores '1. The branch is green. I merge. The result merge commit 'M changes the union of files, so now the tests run, and the commit — now on HEAD — breaks the build. A subsequent unrelated commit M - '3 doesn't modify the relevant code; CI skips the tests and delivers a green result on a broken codebase. People erroneously think "problem fixed". I have seen this all play out in person, multiple times.)¹
(A "much easier" approach is to simply cache a single build step: you hash your inputs, and compute a cache keys; see if your output is cached. If yes use cache, if no build. Computing the cache key is the tricky part and risks that famous "top n problems in computer science … cache invalidation" quote.)
¹while I know how to compute better git diffs, the difference between the common ancestor, the result once the commit gets merged, etc. are subtle. Most devs are shockingly inexperience with git and don't even get this far into the problem, and CI system's insistence on only running on, e.g., "pushes" doesn't help.
Sure. You can't go halfway on monorepos where you check in all the code into one spot but don't build any tooling to manage that. You need to use something like Bazel/Blaze, Buck, or other tools that meant to own responsibility for managing dependencies between projects.
> Well, there aren't "subrepos" … it's all one giant monorepo.
I think op meant organizationally you still have logical components.
> And … no? No, CI tools don't. There's generally not a tool that has the dependency graph, and it's typically not recorded. (Excepting bazel, which set out to solve this problem; lo and behold it was designed by a company with a monorepo, too.)
Everywhere I've worked that has had a monorepo (Google, Facebook), that's definitely the case. The CI automation would query Buck/Bazel to figure out the set of dependencies impacted by a PR. Of course, some PRs would have outsized impacts (e.g. changing libc) but at the same time, there's probably not much better than that.
Apple was a bit different. While nominally each project had it's own git repository, you uploaded code to a central monorepo that organized everything by release. And it built every project incrementally and relinked/rebuilt whichever projects were impacted. They didn't at the time have a centralized CI system. But also Apple's system evolved from many decades ago and is a sane selection for building an OS from back then. Google's approach I think is generally accepted as a more effective strategy in some ways if you're going down that route. That being said, at Google scale you're shipping so much code there's still challenges. For example, there's so much code being changed at Google's scale, that they have to bundle things together into a single CI pass because there's just insufficient compute capacity available to do everything + avoid serialization of unrelated components. Of course, probabilistically there's a non zero chance that something is broken and they intelligently bisect and figure out what change needs to be omitted from the ship. Very complicated. But I think most people underappreciate that they'll never encounter these kinds of problems. If you just go all in on monorepo + Bazel + bazel-aware CI setup + build artifact caching, you're done and don't have to think about builds or code management very much after that. That's a really big superpower.
> I think op meant organizationally you still have logical components.
Yes, my impression was that all of these monorepo tools have a first-class notion of the subrepos/projects/workspaces/whatever that make up the monorepo. If you don’t have that, then I guess I don’t really know what you mean when you say you have a monorepo.
I dont understand. You can have 1 repo with multiple services that can be deployed independently.
Edit: perhaps the difference is that you said "monolith." I guess I'm not sure precisely what you mean by this, but context makes it seem like you're using it synonymously with monorepo. Since that's what this thread is about.
There's a very simple solution to that, if your systems & processes are reasonably lightweight.
Just build and deploy everything on every merge. Compute is fairly cheap, and if running in parallel, it doesn't have to take long.
You can also take it a step further and have "mono" binaries/container images, where you specify the service to execute as the first argument.
I've been doing this for about 5 years now, having a single output artifact for each language being used. It works great.
If you're careful about your optimisations, you can go from hitting the merge button to having 100+ services deployed on production in about 60 seconds
Arguably it's a bit of an extremist approach, but if you have a situation where technically you're deploying thousands of times a day, you get pretty good at making the process reliable and robust
> If you know "this thing is broken", the commit log is a fantastic resource to figure out what changed which may have broke it
git supports both diff and log for specific directories, although this may not help you if the issue was with a dependency in another folder that was updated.
But the point is that the tooling doesn't help you with it. Those are building blocks that you might build a "does this need to get build/deployed?" (& if not, what is the result of the build) mechanism with, but they are not that mechanism.
I agree that the blog point didn't make a very strong argument, but there are some inaccuracies in your comment as well.
With regard to the "only run X tests if X changed" problem, Bazel, Buck, and all the other monorepo build tools do this. I mean sure, if you're using some build system not meant for a monorepo you're going to have a terrible time, but who is really going to spend weeks or months converting to a monorepo and not also switch to Bazel (or something akin to it?) at the same time. In fact, I would say switching to Bazel (or Buck, etc.) for builds is a prerequisite to even starting on the path to switch to a monorepo.
This is just a really useful feature even if you're not in a monorepo. Sometimes you're changing some core header file or whatever and you really do need to run nearly all the tests in your test suite. Sometimes you're just changing some fairly self-contained file and only a few tests need to run. Sometimes you change some docs in the repo and you don't need to run any tests at all. Bazel will just do this automatically for local builds (it knows what tests have transitive dependencies that have changed since the last time those tests were run), and setting it up in CI is a few lines of bash or Python. To set this up in CI you basically just check which files have changed since the last time CI ran (e.g. using git diff), then you use bazel query to find all test targets that have transitive dependencies on those files, then you feed that list of test targets to bazel test. You can set this up per-developer branch for instance, so that if you have a bunch of developers all running tests on the same set of CI machines you get good caching.
With regard to colocating schemas in APIs, yes you can do that but it's really annoying to do with Protobuf/Thrift. First of all protobuf and thrift require that IDLs exist locally so they can do code generation, so if you have protobuf files split into multiple services you need a way to distribute them all which is super annoying. Additionally, in some cases there isn't a clear single owner of a particular IDL struct, for example let's say you have some date or time struct that many protobuf messages want to use in their fields. Which service do you define it in? Ignoring that, it is REALLY USEFUL to be able to modify the code for the producer of a message and the consumer of a message all at once, without having to make multiple commits in multiple repositories. This is especially true when it comes to testing. I have thing A producing a new field X, I want B to use the new field X, and I want to test that B uses it correctly. When everything is in one repo this just works, with multiple repos I need to first add the code to thing A, do a release of A (even if that's just making and pushing a git commit), then update B to consume the new thing and add the test, then if I realize I messed something up I need to go update repo A again to test it, and so on. Obviously this works and tons of people do it, but it sucks. I had to do this at my last job (which wasn't using a monorepo), and it worked but it was cumbersome and I hated it.
With regard to scaling, a lot has changed in git in the last two years to make it possible to run huge git monorepos without any weird hacks. The most notable such feature is sparse indexes, which let you clone a subset of a git repo locally and have it work normally. Here's a GitHub blog post about sparse indexes: https://github.blog/2021-11-10-make-your-monorepo-feel-small... . They also have a monorepo tag which you can use to look at other blog posts about monorepos (and as you'll note, most of these are pretty recent): https://github.blog/tag/monorepo/
The biggest downside of a monorepo in my opinion is that there are a lot of things that Bazel makes way harder than the default package manager for language X. Practically speaking it's probably going to be hard to use Bazel if you don't have a dedicated build team with experts who can spend all the time to figure out how to make Bazel work well in your organization. This is pretty different from just using pip or npm or yarn or whatever, where you can get things just working in a couple of minutes and the work to maintain the build system is probably just collectively a few hours of work a week from people spread throughout the organization who can be on any team. For a small organization I can't see it being worth the effort unless you already have a lot of engineers who have a background in Bazel, for example. But there's definitely a point where the high entry-level cost to Bazel and a monorepo makes sense.
> you need a way to distribute them all which is super annoying
Only if your tools are bad. In the stack I'm used to they're just another artifact that gets published as part of a module build.
> Additionally, in some cases there isn't a clear single owner of a particular IDL struct, for example let's say you have some date or time struct that many protobuf messages want to use in their fields. Which service do you define it in?
A common library, just like any other kind of code.
> Ignoring that, it is REALLY USEFUL to be able to modify the code for the producer of a message and the consumer of a message all at once, without having to make multiple commits in multiple repositories. This is especially true when it comes to testing. I have thing A producing a new field X, I want B to use the new field X, and I want to test that B uses it correctly. When everything is in one repo this just works, with multiple repos I need to first add the code to thing A, do a release of A (even if that's just making and pushing a git commit), then update B to consume the new thing and add the test, then if I realize I messed something up I the new version ofneed to go update repo A again to test it, and so on.
That's actually really important because it forces you to test all the intermediate states. If you can just change everything at once then you will, and you probably won't bother testing everything that actually gets deployed, and so you get into a situation where if you could deploy the new version of A and B at exactly the same time then it would work, but when there's any overlap in the rollout everything breaks horribly.
It really sucks to manage shared libraries, across many clients in a few languages. Any update requires updating the version everywhere, and you have to independently debug which version of which library was being used by which application at the time a bug occurred. It works, it isn't impossible, obviously lots of people manage it successfully (polyrepos are more common than monorepos in my experience), but it's a giant pain and it sucks.
> Any update requires updating the version everywhere
Much of the benefit of using an IDL like this is to be (mostly) forward compatible, so you don't have to upgrade everywhere immediately.
> you have to independently debug which version of which library was being used by which application at the time a bug occurred
You have to do that anyway; it's easier if your repository history reflects the reality of which applications were upgraded and deployed at which times. There's nothing worse than having to debug a deployed system that was deployed from parts of a single repo but at several different points in that repo's history.
There's also challenges with local dev tooling. Many VSCode language extensions, for example, won't operate at their best if the "language signaler" file (e.g. package.json for JS) isn't in the root of the repo; from just refusing to function, to intellisensing code in another project into this one, all manner of oddities.
Meanwhile; I don't think the purported advantages are all that interesting. Being able to write the blog post and make the change in the same PR? I've never been in a role where we'd actually want that; we want the change running silently in prod, publish the blog post, flip a feature flag or A/B test proportions. Even an argument like "you can add schemas and the API changes in one go"; you can do that without a "monorepo", just co-locate your schemas with the API itself, this isn't controversial or weird, this is how everywhere I've worked operates.
None of that is to even approach the scale challenges that come with monorepos at even reasonable sizes; which you can hit pretty quickly if you're, say, vendoring dependencies. Trying to debug something while the Github or Bitbucket UI is falling over because the repo is too large, or the file list is too long, isn't fun.
I'm not going to assert this is a hill I would die on, but I'm pretty staunchly in the "one service, for one deployment artifact, for one CI pipeline, for one repo" camp.