I'm surprised to see that Boolean feature flags are common. We almost always ramp up percentages of UUID space or randomly chosen requests, to reduce the blast radius of a bad change.
At $JOB, we have feature flags of innumerable shapes and sizes. Some are based on account standing, some are % gradual rollout at random, others are a more thoughtful low-risk to high-risk rollout across customers and hosts. Some are manual flipped per customer/only on certain dev hosts. Literally anything you can think of, we have tied behavior to it.
But, we've got good frameworks in place such that at the call sites where behavior diverges, it's just checking a boolean.
if PermissionController.get().get(MyPerm.class): doA() else: doB().
I suspect this is pretty common, and its still easy to do the dead code elimination on.
> if PermissionController.get().get(MyPerm.class): doA() else: doB().
oh, but if a behavior changes when a feature flag is active, there's a very strong case for it to be pluggable behavior strategy, so I like these so much better as an unconditional call to `self.getThingStrategy().execute()`
Yes, but you move the logic for deciding which branch to take to an underlying function. It pollutes the parent function less, but results in more overall functions for places to hide.
Also, if the logic gets more complicated than just an if statement (if Permision... and date < cutoffdate: etc) you don't further pollute the parent function.
Naah, but at least that would make the decision simple and clear (if condition return doA else return doB).
Better would be for whatever ThingFactory or getThingService instantiates the Thing to make the decision, and compute it up front.
If-else statements in application logic tangle the concepts of "what should be done and why?" with "let me do this Way 1" and "let me do this Way 2". Ideally a typical service (or model or similar) shouldn't be aware that "feature flags" as a concept exist, and this should be regarded as inimical to their encapsulation. It should just know that it delegates a decision to Way N.
Many feature flag systems ramp up doing that from the config server side. So when your client requests feature flags, it gets assigned true/false on that basis.
The flag becomes a percentage. The feature isn't always off or always on, it's on for a small fraction of workload, and then that fraction grows as you demonstrate it's safe. Ideally you want metrics that tell you whether the experiment is worse or better than the control group.
What you're describing is a common way of using feature flags—except the percentage part comes from how you manage the servers running the binary with config. I.e. on day one, 5% of servers in cluster get True for the flag value. The double the percentage every day until 100% or otherwise rollback if it's a bad cut.
Then rolling forwards and backwards is a whole deployment away, or mucking about with infrastructure, vs tweaking a percentage flag somewhere.
If you want to get fancy with changes (and I've seen it done) you have something else capable of controlling that percentage setting that is tied in to your monitoring. Start out low, say 1% of requests hitting the new path. Automatically ramp up over time to full 100%. If you see failures, automatically drop back to 0% until it can be ascertained that the failure didn't come from the new code.
Partitioning by instance works if you have enough instances to avoid big increases, but at that point you can just deploy known-good and new-feature builds. Runtime checking helps if it's a lot faster than rolling back to the known-good build, or if you're doing concurrent experiments (you may not have enough instances to try every possible combination).
Having done it both ways this would not be my recommendation unless it's necessary - I think it adds a fair amount of complexity.
Some considerations: you'll need some sort of storage mechanism for these flags - is that a centralized configuration service for all your services? Maybe just a table in your database? But database / network calls are expensive to be adding to every single time your code executes the path in question - maybe it makes sense for your service to cache these values locally...but then doesn't that lose part of the purpose of 'fast rollbacks'? Maybe instead of a local cache you spin up a redis instance - but what if this goes down? Will all your instances default to the same value? Etc, etc, etc.
I'm not saying this approach is bad, only that it has complexity, and I find I generally can get away without it.
I think it might be less confusing to say, how do you verify the feature? As in: how do engineers and product managers and designers know that the flagged behavior is correct and how do you verify that in production if all you do is ramp up? How do you make sure the interested parties are always bucketed into the on experiment?
