>> storing my.cnf in VCS is a better idea than updating it in prod.
Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.
Storing passwords and other secrets in VCS is a phenomenally bad idea, particularly when your team size grows past two or three people.
All it takes is one fuckup of accidentally committing to a public repo. Or firing someone on the team but not thinking to rotate secrets. Or hiring an outside contractor that now you implicitly give access to your production AWS credentials, etc.
Secrets do not belong in source control, hard stop.
I don’t understand why secrets should be excluded from source control. It seems like a perfectly fine place for them to be stored to me especially when talking about secrets which the developers require in order to develop the code or maintain a set of systems.
If the secrets in source control are:
(1) encrypted
(2) never have decryption keys stored/loaded on any developer machine
(3) never have stored decrypted representations (only in memory representation of decrypted forms allowed when required).
If you follow these rules you won’t be more likely to accidentally commit unencrypted versions of the secrets and you’ll also by necessity have setup some (auditable) gatekeeper for logging of decryption events (via an aws kms or similar decryption as a service api).
For the category of secrets which must never exist in any decrypted form on a developer machine — maybe I can see the argument that those should be left out of source control as this would represent a reduction in the surface area for offline attacks against the encrypted form of the secrets -— but I would guess that this actually represents a somewhat minor gain in practice? And also seems strictly unimportant to use cases where access to the decrypted representation of the secrets on the developers machine is mandatory ...
It's so easy to avoid having them in VCS though. Any secrets can be loaded from a configuration file placed in ~. Populated configs can be selectively distributed so you can e.g. only place the production config file (with the prod server AWS keys for example) on the CI server. Everyone else just gets a config file with non-prod secrets. This can help avoid a lot of mishaps.
The problem with config files outside of vcs is keeping them in sync, which leads to bugs, insecure sharing methods, and constant interruptions as developers need to bug each other to get the latest values.
Shameless plug, but check out EnvKey[1].
With 10-15 minutes of effort, it gets secrets out of git, out of email, out of Slack, etc. It manages encryption keys safely behind the scenes, protects development secrets just as strongly as production, keeps devs and servers automatically in sync, and greatly simplifies access control and key rotation.
It's not the only solution out there, but it's by far the easiest to setup and work with. In any case, use something that truly solves the problem! Don't settle for half-measures that end up spraying secrets all over third party accounts. This stuff is serious--even when it comes to so-called development secrets, the line is fuzzy.
Shameless or not, this is the first time I’ve seen a hazmat[1] handler I understood by the end of the first page and the price is low enough I’m just signing up now to use it for all my personal projects. Thank you for sharing it here.
1 - encryption keys and secrets are “hazardous material”, shortened to just “hazmat”. While necessary and arguably crucial in our work, they deserve the same care and respect a chemist would have for a beaker full of particularly dangerous chemicals.
Also, it really depends on the environment, the nature of secrets, etc. It's never great, but the level of badness varies dramatically.
If you have a team of 2-3 people, where everyone has access to everything anyway, and the secrets can't be used from outside the company (i.e. you need access to servers in the first place to authenticate with whatever service is configured), then it's one thing. It's bad, but it sure beats 'let's really carefully update it on all machines manually'.
I just really, really don't like blanket statements such as 'you shouldn't do that, full stop, context is irrelevant'.
Even then, secrets don't even belong in source control. They aren't tied to a specific version of your software -- they're tied to state in other systems: AWS, your MySQL database, a third-party API, etc. Those systems will change independently of any particular release of your code, and having it versioned in source control doesn't actually make sense.
They are tied to the overall state of a application though. Much like db versions, 3rd party apis etc.
You must track them somehow and a VCS is obviously the right thing to store them in.
Depending on the circumstances, I use git-encrypt, Ansible vault, vim -x (encrypt files) or even RCS (no chance of pushing to a repo when RCS doesn't have that feature).
This one of my criticisms of the 12 factor app. They say put the config in the environment. That doesn't address the question. How and why does it get into the environment and from where?
Secrets belong wherever it makes the most sense for them to be, balancing operational needs, security, and compliance requirements.
Storing secrets in version control (with permissions to the repo tightly controlled), encrypted (with the decryption key only available on the compute needing secret access), and used for rendering into a discovery mechanism (or directly injected into instance env vars) is entirely legitimate (and I know of several orgs doing this).
It is wholly possible to store configuration data in vcs in a way that doesn't sacrifice depth of security when it comes to secrets and passwords.
But otherwise yes, storing passwords in vcs is bad. That wasn't necessarily what I meant to imply by suggesting storing configuration files in vcs, however. I probably phrased that poorly.
I get you, wrong wording, it's not exactly a standard unit test, etc. But it doesn't change the meaning and intention of it - treat infrastructure as you treat code (I guess that also includes debating whether something is a 'unit' test or 'other' kind of test).
Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_ of a command or something else.
>Also, having a file on a container (which is probably the easiest test to perform) often is _the desired behavior_
No it isn't. The end user doesn't give a damn whether a particular file is on the container. That's an implementation detail. The end user wants:
* Pages to load quickly
* To not have to face data inconsistency bugs
* Pages to operate while the site is under high load
* For various services your system connects to to work properly (e.g. clicking 'get two factor code' actually sends an SMS).
Checking to see if a certain file is present is pointless if that doesn't lead to the desired system behavior.
Moreover, if you have the means to verify the desired system behavior:
* The presence of the file, if it was required, can be assumed.
* If you swap out a component and stop needing that file to be present to achieve the same behavior your test will still fail even if your system works perfectly. That's an extremely undesirable property to have in a test.
Checking for the presence of a file and failing hard if it is not there as part of a build is a sometimes a good way sanity checking a component, but as an outside 'test' of that component it's a bad idea.
In which case it would make most sense to build a mock container on top which clearly exhibits the desired behavior of the underlying container and then test the behavior of that.
Maintainer of the tool here - I'm not sure I understand this idea. What would go in the mock container? How would you test against it? Would you mind elaborating a little more?
I'm assuming that in this case the base container would be a 'ruby' container or something. You'd build an example container atop it running a bit of example ruby code - verifying that it behaves properly. Those behavioral tests could be used to verify that the underlying container is configured properly.
I suppose that's sort of what you're doing in the example you linked to above.
I guess an "example". I generally agree with the parent poster. Testing for files does not sound useful. Test the functionality, not structurue. But not everyone's google.
My pet peeve with all these time trackers is that... It's just a wrong metric, unless what you're doing is an assembly line equivalent, where you just need to put in the hours.
And don't get me started on such trackers having 'leaderboards' where you're 'competing' with colleagues.
Good to see this. I think that hiring remotely and paying Bay Area salaries (or very close to it) does indeed open up access to talent that might be harder to reach otherwise. Especially, if hiring globally remote.
I've been working remotely from home for the past 7 months, I live in Vilnius, Lithuania (so the local job market is way more limited than in SF, London, etc).
I think that it largely depends on the company. Because remote work doesn't Just Work -- it requires plenty of regular effort to make that happen. Some companies excel at that, and others, well, suck at it. So if you consider doing, ask a lot. Ask a lot about processes, how decisions are made, why the company decided to go remote, etc. There, you can't really generalize everything under 'remote work is good for you' or similar. Just like working from an office doesn't automatically mean that it's good for your career.
If the company went the remote path in order to access untapped talent from around the globe, communication is heavily written and async, that's great. Ask for specific examples, as pretty much everyone says 'yeah, we are all about written communication!'. If the company provides money to setup your own private home office for maximum comfort and productivity -- even better. That shows that they really care. After talking to a few companies and comparing them, it's easy to tell where a company falls on the 'good for remote' spectrum.
Is it sustainable long term (7+ years) is a whole different matter and I don't have an answer to it. Probably completely depends on the person.
To answer your questions
> Especially if the team is a mixture of remote and co-located people?
I work in such environment and I'm very happy. But I understand how in some cases it can end up really badly. So just do your research.
> How have you tried to mitigate that?
Just do your best to pick the right company. Because if you join one which isn't great for remote work, then it's unlikely that you'll be able to change how it works.
This is my personal experience so far:
Pros:
* More available jobs, especially for something which is not full-stack web-dev (SRE side of big data/analytics is my niche). Which is great for developing my technical skills and adding some lovely things to my CV;
* I feel overall more positive about work and life in general, which is a result of good time flexibility, not having to commute anywhere, setting up a solid home office, etc. While not directly related to career, this huge! It reduces chances of burning out and similar negatives as well.
The main and very obvious con is non-existing serendipitous networking with people. But that's largely due to my own choices - I could go to a co-working space, etc but I chose not to. Living with my gf helps with having decent social life. That works for me.
So all things considered, that's a solid net positive.
I've tried both Google and HERE maps when traveling to a number of countries, and found HERE significantly better offline. It was constantly able to detect my exact position and update it as I kept walking, while Google Maps were quite inaccurate.
This whole thing looks rather silly. Like, you either find a decent exchange, expect it to be legit and that includes cooperating with IRS and other parts of the government when needed. You know, like you expect responsible financial institutions to do. Or just find some maybe less legit ones at your own risk.
Coinbase can't be both legit and 'oh look we resist The Man'. Urgh.
If meeting in person is not convenient enough- well, this is really just the tradeoff you make choosing between Coinbase and Craigslist. Convenience vs security/anonimity.
That applies to other kinds of reliability testing too, especially everything that goes through a network.
It's easy to handle when something is completely unavailable (i.e. instant error), but when something, be it a database or some endpoint, is available but horribly slow, that's a whole different thing.
Seems like we need systems designed with "suicide" mechanisms built in, so that if they detect that they have a poor quality of life (err, I mean, that they're providing a poor quality of service) they'll shut down completely.
The circuit-breaker pattern is commonly used to make the affected system appear off-line to it's upstream dependents - you don't need to actually shut the server down completely and will probably want to log into it to see what's happening. If the dependent system can't function without the affected system, it should also indicate an error and further upstream systems should detect it.
In your house, you have the circuit-breaker pattern implemented in hardware. And upstream of your house there are many more layers of circuit breakers that generally increase in size until they reach a point where there is redundancy. A circuit-breaker going off in your house protects the other circuits in your house, a circuit-breaker going off on your street protects the rest of your neighborhood.
Industrial circuit-breakers are commonly a combination of hardware and software. Personally, I've lost more equipment due to brown-outs than any other cause. If you have equipment you don't want to experience a brown-out, program the breaker to cut off power completely.
Eh, 'but we can't store things/API keys/passwords in VCS! It's bad!' is a pet peeve of mine. No, let's not 'just update it manually when we need it'. No, if outsider getting your DB password (somehow) poses existential threat... Chances are that you have way more concerning problems.