Skip to main content

5 posts tagged with "CI/CD"

View All Tags

· 4 min read
Rafael Gonzaga

Supply chain attacks are not something new; we have heard about them extensively, and the maximum we can do is mitigate them as best as we can. However, it is crucial to acknowledge that these types of attacks will always exist. With that in mind, it is important to understand all the attack vectors and take the necessary steps to secure our environment.

One of the initiatives planned by the Node.js Security WG (Working Group) for 2023 is to enhance the OSSF Scorecard. This task requires changing all Node.js actions to be pinned by commit-hash. The reason for this approach is quite simple: commit-hash provides immutability, unlike tags which do not.

For instance, it is quite common to include the following action as part of our application's CI pipeline:

jobs:
build:
name: Build, push
runs-on: ubuntu-latest
steps:
- name: Checkout master
uses: actions/checkout@v3.5.2

Many developers rely on tools like Dependabot or Renovatebot to ensure that these actions stay up-to-date. However, using the release tag can pose a risk to our environment.

Looking at a Scenario Where a Malicious Actor Gets Control

Let's consider a scenario where a malicious actor gains control over the actions/checkout package. This compromised package can now potentially manipulate the entire CI process. It can access environment variables used by other jobs, write to a shared directory that subsequent jobs process, make remote calls, inject malicious code into the production binary, and perform other malicious activities

What many developers assume is that once they pin an action using a release tag, such as v3.5.2, they are safe because any new changes would require a new release. However, this assumption is fundamentally incorrect. Release tags are mutable, and a malicious actor can override them. To illustrate this point, I have created two repositories for educational purposes:

  1. bad-action - This repository contains a GitHub action that simulates someone taking over the package.
  2. using-bad-action - This repository demonstrates a project that utilizes the aforementioned action, as the name suggests.

In the .github/workflows/main.yml file of the latter repository, the bad-action is being used in version v1.0.1:

on:
workflow_dispatch:

jobs:
example_job:
runs-on: ubuntu-latest
steps:
- uses: RafaelGSS/bad-action@v1.0.1

For this practical example, workflow_dispatch will be used, but the same applies to on: [push, pull_request] processes and so on.

As a result, when the action is executed, it prints “Hello world” in the console.

Now, let's consider the scenario where a bad actor takes over the repository and modifies the "Hello world" message to "Hello darkness my old friend" without creating a new release. Instead, the actor overrides the existing v1.0.1 release using the following commands:

echo "echo \"Hello darkness my old friend\"" > run.sh
git add run.sh
git commit -m "dangerous commit"
git push origin :refs/tags/v1.0.1
git tag -fa v1.0.1
git push origin main --tags

Consequently, if the action is executed again without any changes made to the source code, it will print "Hello darkness my old friend". This demonstrates how our environment can be exploited by manipulating release tags.

Solution

Pinning an action to a full-length commit SHA is currently the only method to ensure the use of an action as an immutable release.

Quoting the OSSF Scorecard

Pinned dependencies help reduce various security risks:

  • They guarantee that checking and deployment are performed with the same software, minimizing deployment risks, simplifying debugging, and enabling reproducibility.
  • They can help mitigate compromised dependencies from compromising the project's security. By evaluating the pinned dependency and being confident that it is not compromised, we can prevent the use of a later version that may be compromised.
  • Pinned dependencies are a way to counter dependency confusion (or substitution) attacks. In these attacks, an application uses multiple feeds to acquire software packages (a “hybrid configuration”), and attackers trick the user into using a malicious package from an unexpected feed.

With that in mind, fixing or securing the action is a straightforward process:

on:
workflow_dispatch:

jobs:
example_job:
runs-on: ubuntu-latest
steps:
- uses: RafaelGSS/bad-action@e20fd1d81b3f403df56f5f06e2aa9653a6a60763 # v1.0.1

Adding a comment referring to the tag enables automated dependency updates, such as Dependabot or Renovatebot, to update it whenever a new release is detected. We can see an example of this in action in the Node.js Security WG repository.

There are open-source tools like StepSecurity that can assist us in addressing these concerns. It generates automated pull requests for our codebase based on the configuration specified on their website.

· 2 min read
Jiaqi Liu

Overview

Being a strong proponent of Immutable Infrastructure, [hashistack] is constantly pushing the limits of its ability in various use cases, one of which is the Configuration Management

Traditional configuration management includes Chef, Puppet, and Ansible. They all assume mutable infrastructure being present. For example, Chef has a major component responsible for jumping into a VM, checking if config has been mutated before apply any operations.

With the adoption of Immutable infrastructure, we initially stored and managed our configuration, such as SSL certificate or AWS SECRET ACCESS KEY directly in GitHub Secrets. This has the disadvantage of not being able to see their values after creation, making it very hard to manage.

Then we moved to a centralized runbook, where everything can easily be seen and modified by authorized team members. In this approache, CI/CD server will pull down the entire runbook and simply pick up the config files. This, however, exposed a great security risk because illegal usage could simply leak any credentials to public by cating that credential file out

So the problem, or what [hashistack] is trying to solve here, is

  • being able to keep credentials, whether it's string values or values stored in files, secure, and
  • allowing team member to easily manage those credentials
note

We tried HashiCorp Vault but it doesn't support storing file credential, [hashistack] addressed exactly how file can be managed in this case

So this brought us to the alternative way of thinking about Configuration Management in Immutable Infrastructure, which is depicted below:

We still need GitHub Secrets because our tech dev has a deep integratin with it and that's the most secure way to pass our organization credentials around.

In addition, we will also keep runbook for config management. The runbook will be hosted separately, not in GitHub Secrets.

info

Runbooks was used in Yahoo that keeps all DevOps credentials in a dedicated GitHub private repo. It's been proven to be an effective way to manage and share a software configurations within a team.

hashistack's github-secret now comes into play to bridge the gap between two componet.

· 8 min read
Jiaqi Liu

Trunk-based development is a version control management practice where developers merge small, frequent updates to a core "trunk" or main branch. Since it streamlines merging and integration phases, it helps achieve CI/CD and increases software delivery and organizational performance.