The problem is that in microservice environments, a lot of complexity and source of bugs are (hidden) in the complex interactions between different components.
I also believe that this mentality of siloing/compartmentalization and habit of throwing things over the fence leads to ineffective organization.
After close to a decade of working in various microservice based organizations, I came to a big-ish monolith project (~100 devs). Analyzing bugs is now fun, being able to just step through the code for the whole business transaction serially in a debugger is an underrated boost. I still need to consult the code owners of a given module sometimes, but the amount of information I'm able to extract on my own is much higher than in microservice deployments. As a result, I'm much more autonomous too.
> Maybe code changed. Maybe someone kubectl edit'ed something manually. Not everything is represented in code.
That's honestly one of the big problems in microservices as well.
> After close to a decade of working in various microservice based organizations, I came to a big-ish monolith project (~100 devs). Analyzing bugs is now fun, being able to just step through the code for the whole business transaction serially in a debugger is an underrated boost. I still need to consult the code owners of a given module sometimes, but the amount of information I'm able to extract on my own is much higher than in microservice deployments. As a result, I'm much more autonomous too.
Could you expand how do you manage ownership of this monolith? Do you run all the modules in the same fleet of machines or dedicated? Single global DB or dedicated DB per module (where it makes sense, obviously)?
Because where I work we have a big monolith with a similar team size and it's a royal PITA, especially when something explodes or it is going to explode (but we have a single shared DB approach, due to older Rails limitation, and we have older Rails because it is difficult to even staff a dedicated team that take care of tending the lower level or common stuff in the monolith).
> Could you expand how do you manage ownership of this monolith?
We have a few devops teams (code + deployment) and platform teams (platform/framework code), the remaining teams (which form the majority of devs) own various feature slices. The ownership is relatively fluid, and it's common that teams will help out in areas outside of their expertise.
> Do you run all the modules in the same fleet of machines or dedicated?
Not sure if I understand. All modules run in the same JVM process running on ~50 instances. There are some specialized instances for e.g. batch processing, but they are running the same monolith, just configured differently.
> Single global DB or dedicated DB per module (where it makes sense, obviously)?
There is one main schema + several smaller ones for specific modules. Most modules use the main schema, though. Note that "module" here is a very vague term. It's a Java application which doesn't really have support for full modules (neither packages nor Java 9 modules count). "module" is more like a group of functionality.
> and we have older Rails because it is difficult to even staff a dedicated team that take care of tending the lower level or common stuff in the monolith).
This is usually a management problem that they don't pay attention to technical debt and just let it grow out of control to the point where it's very difficult to tackle it.
The critical part of the success of this project is that engineering has (and historically had) a strong say in the direction of the project.
But aren't micro-service specifically designed to be able to split responsibility of a large system between multiple teams. If everybody debugs and fixes bugs across the whole landscape, than everybody has to be familiar with everything, which means you are loosing the benefits. Occasionally, it might be helpful to debug the whole stack at once. But I wouldn't trust a landscape where that is needed too often. I might be that the chosen abstractions don't fit well.
> But aren't micro-service specifically designed to be able to split responsibility of a large system between multiple teams.
That's the idea, but business transactions usually span multiple services and bugs often aren't scoped to a specific service.
> If everybody debugs and fixes bugs across the whole landscape, than everybody has to be familiar with everything
A lot of things can be picked up along the way while you're debugging, and I'm usually able to identify the problem and sometimes even fix it.
> I might be that the chosen abstractions don't fit well.
Very often the case. Once created, services remain somewhat static, their purpose and responsibility often gets muddy. Mostly because "refactoring" microservice architecture is just very expensive and work intensive. Moving code between modules within a monolith is rather easy (with IDE's support), moving code between services is usually not trivial at all.
I also believe that this mentality of siloing/compartmentalization and habit of throwing things over the fence leads to ineffective organization.
After close to a decade of working in various microservice based organizations, I came to a big-ish monolith project (~100 devs). Analyzing bugs is now fun, being able to just step through the code for the whole business transaction serially in a debugger is an underrated boost. I still need to consult the code owners of a given module sometimes, but the amount of information I'm able to extract on my own is much higher than in microservice deployments. As a result, I'm much more autonomous too.
> Maybe code changed. Maybe someone kubectl edit'ed something manually. Not everything is represented in code.
That's honestly one of the big problems in microservices as well.