Big fan of GitLab (and GitLab CI in particular). I had a recent requirement to push changes to a wiki repo associated with a GitLab project through a GitLab CI pipeline (using the SaaS version of GitLab) and ran into a conundrum…
Using the GitLab SaaS version – deploy tokens can’t have write api access, so the next best solution is to use deploy keys, adding your public key as a deploy key and granting this key write access to repositories is relatively straightforward.
This issue is when you attempt to create a masked GitLab CI variable using the private key from your keypair, you get this…
I was a bit astonished to see this to be honest… Looks like it has been raised as an issue several times over the last few years but never resolved (the root cause of which is something to do with newline characters or base64 encoding or the overall length of the string).
I came up with a solution! Not pretty but effective, masks the variable so that it cannot be printed in CI logs as shown here:
Add a masked and protected GitLab variable for each line in the private key, for example:
Add the following block to your .gitlab-ci.yml file:
now within Jobs in your pipeline you can simply do this to clone, push or pull from a remote GitLab repo:
as mentioned not pretty, but effective and no other cleaner options as I could see…
With the adoption automation for deploying and managing application environments, protecting privileged accounts and credential secrets in a consistent, secure, and scalable manner becomes critical. Secrets can include account usernames, account passwords and API tokens. Good credentials management and secrets automation practices reduce the risk of secrets escaping into the wild and being used either intentionally (hacked) or unintentionally (accident).
Reduce the likelihood of passwords slipping into source code commits and getting pushed to code repositories, especially public repositories such as github.
Minimise the secrets exposure surface area by reducing the number of people who require knowledge of credentials. With an automated credentials management process that number can reach zero.
Limit the useful life of a secret by employing short expiry times and small time-to-live (TTL) values. Automation enables reliable low-effort secret re-issue and rotation.
The following objectives have been considered in designing a secrets automation solution that can be integrated into an existing CICD environment.
Integrate into an existing CICD environment without requiring an “all or nothing” implementation. Allow existing jobs to operate alongside jobs that have been converted to the new secrets automation solution.
A single design that can be applied across different toolchains and deployment models. For example, deployment to a Kubernetes environment can use the same secrets management process as an application installation on a virtual machine. Similarly, the design can be used with different CICD tools, such as GitLab-CI, Travis-CI, or other build and deploy automation tool.
Multi-cloud capable by limiting coupling to a specific hosting environment or cloud services provider.
The use of secrets (or not) can be decided at any point in time, without requiring changes to the CICD job definition, similar to the use of feature flags in applications.
Enable changes to secrets, either due to rotation or revocation, to be maintained from a central service point. Avoid storing the same secret multiple times in different locations.
Secrets organised in predictable locations in a “rest-ish” fashion by treating secrets and credentials as attributes of entities.
Use environment variables as the standard interface between deployment orchestration and deployed application, following the 12 Factor App approach.
Secrets stored centrally in Hashicorp Vault.
CICD jobs retrieve secrets from Vault and configure the application deployment environment.
Deployed applications use the secrets supplied by CICD job to access backend services.
Use Vault by Hashicorp as a centralised secrets storage service. The CICD service retrieves secrets information for integration and deployment jobs. Vault provides a flexible set of features to support numerous different workflows and available as either Vault Open Source or Vault Enterprise. The secrets management pattern described uses the Vault Open Source version. The workflow described here can be explored using Vault in the unsecured development mode, however, a properly configured and managed Vault service is required for production use.
Vault supports a number of secrets backends and access workflow models. This solution makes use of the Vault AppRole method, which is designed to support machine-to-machine automated workflows. With the AppRole workflow model human access to secrets is minimised through the use of access controls and temporary credentials with short TTL’s. Within Vault, secrets are organised using an entity centric “rest-ish” style approach ensuring a given secret for a given service is stored in a single predictable location.
The use of Vault satisfies several of the design objectives:
enables single point management of secrets. The secrets content is stored in a single location referenced at CICD job runtime. On the next invocation, the CICD job retrieves the latest version of the secrets content.
enables storing secrets in predictable locations with file system directory style path location. The “rest-ish” approach to organising secret locations enables storing a given secret only once. Access policies provide the mechanism to limit CICD visibility to only those secrets required for the CICD job.
Passing Secrets to Applications
Use environment variables to pass secrets from the CICD service to the application environment.
There are existing utilities available for populating a child process environment with Vault sourced secrets, such as vaultenv or envconsul. This approach works well for running an application service. However, with CICD, often there are often sets of tasks that require access to secrets information as opposed to a single command. Using the child environment approach would require wrapping each command in a CICD job step with the env utility. This works against the objective of introducing a secrets automation solution into existing CICD jobs without requiring substantial refactoring. Similarly, some CICD solutions such as Jenkins provide Vault integration plugins which pre-populate the environment with secrets content. This meets the objective of minimal CICD job refactoring, but closely couples the solution to a particular CICD service stack, reducing portability.
With a job script oriented CICD automation stack like GitLab-CI or Travis-CI, an alternative is to insert a job step at the beginning of a series of CICD tasks that will populated the required secret values into expected environment variables. Subsequent tasks in the job can then execute without requiring refactoring. The decision on whether to source a particular environment variable’s content directly from the CICD job setup or from the Vault secrets store can be made by adding an optional prefix to environment variables to be sourced from the Vault secrets store. The prefixed instance of the environment variable contains the location or path to the required secret. Secret locations are identified using the convention /<vault-secret-path>/<secret-key>
enables progressive implementation due to transparency of secret sourcing. Subsequent steps continue to rely on expected environment vars
enables use in any toolchain that supports use of environment variables to pass information to application environment.
CICD job steps not tied to a specific secrets store. An alternative secrets storage service could be supported by only requiring modification of the secret getter utility.
control of whether to source application environment variables from the CICD job directly or from the secrets engine is managed at the CICD job setup level as opposed to requiring CICD job refactoring to switch the content source.
continues the 12 Factor App approach of using environment variables to pass context to application environments.
An example workflow for a CICD job designed to use environment variables for configuring an application.
The following are available in the CICD environment.
A job script oriented CICD automation stack that executes job tasks as a series of shell commands, such as GitLab-CI or Jenkins Pipelines.
A secrets storage engine with a python API, such as Hashicorp Vault.
Add a Vault appRole linked to the new policy. This example specifies a new appRole with an secret-id TTL of 60 days and non-renewable access tokens with a TTL of 5 minutes. The CICD job uses the access token to read secrets.
The helper script get-vault-secrets-by-approle could be executed and sourced in a single step, e.g. source $(get-vault-secrets-by-approle). However, when executed in a single statement all script output is processed by the source command and script error messages don’t get printed and captured in the job logs. Splitting the read and environment var sourcing into 2 steps aids in troubleshooting.
Add CICD job vars for Vault access
In the CICD job configuration add Vault access environment variables.
In the CICD job configuration remove the previously used FAKE_APP_PASSWORD variable.
Execute the CICD job
Kick off the CICD job. Any CICD job configuration variables prefixed with “V_” results in the addition of a corresponding environment variable in the job execution environment with content sourced from Vault.