The pitfalls and options of this frequent GitOps apply as you progress your deployments to manufacturing.
GitOps is the apply of representing system configuration in a Git repository after which utilizing Git workflows to handle modifications to that configuration and updates to the system.
That strategy of representing system configuration in a repository is initially easy, seeding configuration information, copying declarative requests from product documentation, and possibly even the occasional scripted sequence.
At first, you’re feeling progress is inevitable, and success awaits a couple of commits across the nook, then you definately hit the last word GitOps foil: secrets and techniques.
Why plain secrets and techniques are unhealthy
Whereas it could be evident that inserting plain credentials on a Git repository will not be the perfect thought, it’s nonetheless price spelling out why it’s problematic, lest somebody feels it could be a suitable compromise when utilizing a non-public repository:
The individuals who work with the GitOps repository might not be the identical individuals licensed to handle the goal environments.
Let’s say you’re the approver of a pull request and wish the resident community professional to assessment modifications associated to the firewall. Unaware that credentials are saved in plain textual content within the repository, you ask the repository supervisor so as to add that professional to the checklist of customers. Out of the blue, the community professional has entry to a buyer database full of personal information. If that individual will not be cleared for that stage of entry, you’re looking in any respect kinds of paperwork and remediation procedures to rotate and deploy new credentials.
Now, let’s assume a greater situation, the place you might be conscious of the credentials within the repository, thus avoiding the unintended disclosure: now it’s essential go outdoors the pull request workflow to contain that individual within the assessment course of.
When the higher situation is inefficiency and the worst situation is akin to juggling knives blindfolded, it’s time to transfer on to raised practices.
A sealed answer
The thought of a sealed secret in GitOps is to encrypt secrets and techniques earlier than including them to the repository, sharing the personal encryption key with those that want to make use of these secrets and techniques. Usually, these encryption keys are positioned on the goal system and utilized by an area agent to decrypt the credentials and place them wherever they must be within the goal atmosphere.
This system permits individuals to work with the repository with out the danger of by chance disclosing credentials, and that’s an enchancment over storing credentials in plain sight.
That method is intelligent, and I used to be fairly keen on it at the start of my journey into GitOps. The preliminary setup is considerably straightforward, the repositories usually are not broadly used each day but, and it’s straightforward to seek out others at that very same stage of adoption vouching for the method, so it’s also possible to discover assist locally.
Getting previous the preliminary stage and going into bigger and extra everlasting deployments, that simplicity gave approach to limitations and dangers, and I deserted the apply altogether. Within the subsequent sections, I spotlight the primary causes you most likely ought to abandon it too.
Purpose #1: The keys to which atmosphere?
Let’s say you’ve gotten a deployment pipeline with a development of “dev,” “take a look at,” “stage,” and “manufacturing” environments. Every atmosphere shall be working the identical software program, however for those who use good SecOps practices, they are going to use totally different units of credentials.
Some GitOps repositories are designed to have one subtree per goal atmosphere, whereas others are designed with a single parameterized tree deployed throughout totally different environments.
Within the case of a folder tree per goal atmosphere, the git repository should have separate areas for the keys for every atmosphere. If you work with the ever present presence of Kubernetes clusters within the enterprise, you’ll be coping with particular person “Secret” assets unfold throughout a number of namespaces, making the sprawl of folders and information unavoidable.
If we have a look at parameterized bushes, then the state of affairs turns into a bit higher, with all secrets and techniques for every atmosphere getting concentrated right into a single file per atmosphere.
No matter how you design the GitOps repository, utilizing sealed secrets and techniques generates extra folders and information, which will increase the quantity of knowledge individuals want to soak up, and the quantity of artifacts the supply pipelines have to deal with.
Purpose #2: The secrets and techniques are … proper there.
Sure, they’re encrypted, and it takes an encryption key to get to the precise credentials, however they’re nonetheless within the arms of probably unhealthy actors. Think about telling somebody how their database credentials are all seen to the world after which continuing to dissuade their nervousness by explaining how the unhealthy actors nonetheless don’t have the important thing.
It’s possible you’ll chime within the feedback part and clarify the technical motive why that is an unfounded concern, however anybody who works in safety will let you know that the psychological facet can also be a part of making clients really feel secure with their selections.
On the extra technical aspect, I had individuals argue that having sealed secrets and techniques within the open is not any totally different than utilizing public key pairs to encrypt visitors, however these are uneven key pairs the place you by no means have the personal key out in public, encrypted, or in any other case.
Lastly, whereas the key itself is encrypted, the metadata round them isn’t. Dangerous actors can exploit committer data to seed social engineering exploits, infer the rotation insurance policies for the infrastructure parts, decide whether or not secrets and techniques are reused throughout environments, and collect many different clues that may help assaults in opposition to the goal atmosphere.
Retreating from all strains of protection in opposition to cyber assaults and pinning all hopes on defending a single level of failure is a horrible start line for a safe system.
Purpose #3: The important thing to safe all keys remains to be a key.
The lifecycle of secrets and techniques within the repository might differ relying on what they’re securing. A database credential might expire each 30 days, whereas a cluster credential might expire each 60 days.
What in regards to the grasp encryption key for the sealed secrets and techniques themselves? Safety insurance policies will finally power you to rotate that grasp key, which would require a separate course of to redistribute the brand new key securely to all goal environments.
However wait! Not having a separate course of to distribute keys to focus on environments was the explanation you selected to seal secrets and techniques within the Git repository within the first place. One might argue dealing with one grasp secret is healthier than dealing with a number of secrets and techniques, however the price of managing one key or a number of keys is just about the identical, with the added problem of managing the sealed secrets and techniques your self and nonetheless needing a password supervisor of some kind to safe and distribute the grasp keys.
Purpose #4: There are higher options.
Git repositories weren’t designed with key administration in thoughts. They don’t have any assist for key rotation, no assist for serving secrets and techniques as symbolic references, no approach to carry out utilization audits, no assist for various ranges of entry to directors, and so forth.
A key administration answer is designed to handle all these necessities, scale back the floor space for potential leaks, and provide mitigation paths in case a secret’s ever compromised.
That discount of floor space is particularly vital, since you can’t by chance disclose or lose a key if it by no means leaves the system. As one instance, on the planet of Kubernetes, clusters are invariably colocated in a service airplane that incorporates a number of key administration options. It is not uncommon for IaaS suppliers to supply backend integration between companies the place key values by no means have to go away the atmosphere.
Sticking with the instance of Kubernetes, the place you might be prone to be utilizing ArgoCD or Flux to your GitOps practices, they at present lack native integration with key administration companies, however they expose extension factors to realize that integration. As an example, the ArgoCD documentation says, “Argo CD is un-opinionated about how secrets and techniques are managed”, however then proceeds to supply a protracted checklist of options to combine with devoted companies.
A extra promising method is getting began by way of the “External Secrets Operator” project, which synchronizes secrets and techniques from varied key administration companies into native secrets and techniques in a Kubernetes cluster (credit score to my colleague Carlos Santana for that reference.)
There could also be legitimate causes to make use of sealed secrets and techniques. Nonetheless, I’ve but to see one framed in a constructive mild past sealed secrets and techniques being “adequate”, which means they’re cheaper to deploy than a correct key administration answer. I not often see the dialogue get into the concerns of every part else concerned in dealing with these secrets and techniques.
I don’t doubt the engineering prowess of these getting down to mimic a key administration service with textual content information in a code repo, however I query the cost-effectiveness of these approaches. The DIY crowd should resort to a mix of inserting information within the git repo, mapping out key rotation cycles to git pull requests, and instrumenting steady deployment pipelines with decryption keys to parse the repo’s contents. And in case you are asking find out how to handle these grasp keys, it’s possible you’ll end up in a relentless cycle of arising with creative ways of making Git act as a key management service.
One can at all times argue in favor of a build-as-we-grow method, however that will place sealed secrets and techniques as a stepping stone in the direction of utilizing a key administration service, and that’s not the case. Making an attempt to “develop out” of utilizing sealed secrets and techniques means modifications to the GitOps backend of selection and re-training operations individuals to utterly change how they deal with credentials. There is no such thing as a pure development, simply paying twice for a similar outcomes.
Eschew sealed secrets and techniques, begin your GitOps apply proper, and use a managed key service.
(Replace on 5/23: In case you like this subject, I wrote a new story together with a few different issues to keep away from.)