Detection Engineering advanced

Your CI/CD Pipeline Is an Unmonitored Production Environment

XZ, tj-actions, Shai-Hulud, Codecov all owned the build, not the code. Scanning is blind to a poisoned build in flight. Detect the runner instead.

April 25, 2026 · 16 min read · Gowthamaraj Rajendran

Look at the four supply-chain incidents that actually mattered in the last two years, and one thing lines up across all of them. The XZ Utils backdoor (CVE-2024-3094, CVSS 10.0) planted its payload during the build: the malicious code lived in the release tarball’s build scripts, not in the Git tree anyone reviewed [1]. The tj-actions/changed-files compromise (CVE-2025-30066) ran inside roughly 23,000 repos’ GitHub Actions runners, dumping credentials out of runner process memory into public workflow logs [2]. The Shai-Hulud npm worm executed during npm install, ran TruffleHog against the build host and CI environment, and pushed the harvested secrets to attacker-controlled GitHub repositories [3]. Codecov’s Bash Uploader, years earlier, exfiltrated the environment variables of thousands of customers’ CI systems [4].

None of these owned the code. They owned the build. And the build (the runner, its process tree, its network egress, the tokens it mints and spends) is a production environment with root-equivalent reach into your cloud, your artifact registry, and your signing keys. Almost nobody instruments it like one.

The industry’s answer to supply-chain risk has been more scanning: SBOMs, SCA, dependency graphs, provenance attestation. All useful. All structurally blind to a poisoned build in flight, because a malicious step running inside a legitimate job does not show up in a manifest of what you declared you would install. This post is about the surface scanning cannot see, and what to actually detect on it.

The Build Runs Your Most Privileged Code, Unwatched

Think about what a CI job is allowed to do. It checks out source, pulls hundreds of transitive dependencies, executes arbitrary code from every one of them, reads secrets out of the environment, assumes cloud roles, and pushes signed artifacts to a registry that production pulls from. If an attacker described that as a target (“a host that runs untrusted code with credentials to your cloud and your release channel”) you would put an EDR agent on it and route its telemetry to the SOC on day one.

You almost certainly have not, because CI feels like infrastructure plumbing rather than a production tier. OWASP’s Top 10 CI/CD Security Risks names both halves of this gap explicitly: Poisoned Pipeline Execution (CICD-SEC-4), an attacker getting malicious code to run inside your pipeline, as the exploited class, and Insufficient Logging and Visibility (CICD-SEC-10) as the reason you cannot investigate it when it happens [5]. The two feed each other. The pipeline runs untrusted code by design, and you keep no behavioral record of what that code did.

This is the same category error I keep coming back to. Your detections themselves live in exactly this environment: a detection-as-code pipeline is a set of runners with commit access to what your SIEM alerts on. The place you trust to prove your other detections work is the place with the least detection of its own.

Scanning does not close this because scanning answers a different question. SCA tells you which known-vulnerable versions you declared. It cannot tell you that a legitimately-versioned, un-CVE’d dependency ran a post-install script that shelled out to TruffleHog and opened a socket to an IP in a country you have never deployed to. That is not a composition question. It is a behavior question, and behavior questions are answered by telemetry, not manifests. This is the coverage-versus-reality gap from The Detection Funnel in a new location: a green SCA dashboard is a coverage number for a threat model the attack does not live in.

Prevention First, And It Shapes What You Detect

Before a single detection: harden. I lead with prevention here for the same reason I do everywhere: the control you enforce is what turns an attacker’s next move into a high-signal event. Detection is the net for what prevention cannot cover, not a substitute for the hardening you skipped.

The build-time hardening that matters, roughly in order of leverage:

Pin actions and dependencies to full commit SHAs, not tags. The tj-actions attack worked by retargeting tags: v45 and friends were moved to point at a malicious commit. A workflow that pinned tj-actions/changed-files@<sha> never pulled the poisoned version. Tag references are mutable; SHAs are not.
Ephemeral, single-use runners. A runner that is destroyed after one job cannot be a persistence host. It shrinks the blast radius of a compromise to one build.
Least-privilege, short-lived tokens via OIDC. Stop storing long-lived cloud keys in CI secrets. Let the runner mint a short-lived token scoped to exactly the repo and branch that should be deploying, which, not incidentally, is what makes the OIDC exchange auditable later. CI/CD OIDC tokens are workload identities, with all the governance that implies; I made that case in Workload Identities Have a Home Address, and the build runner is where those identities are born.
SLSA provenance. Generate signed provenance so consumers can verify how an artifact was built. SLSA Build L3 requires a hardened platform that isolates runs from each other and protects signing material from user-defined build steps [6].

Every one of these directly improves a detection outcome. SHA-pinning means any pull of a non-pinned or newly-changed action is now anomalous rather than normal. Ephemeral runners mean a runner that outlives its job, or reaches back out to the internet after its build step finished, is worth an alert. OIDC means every privileged cloud action carries a verifiable claim about which repo and branch made it: a claim you can baseline and alert on when it drifts. Prevention is not the opposite of detection. Done right, it manufactures your cleanest signals.

The Strongest Counterargument: “You Can’t Detect Your Way Out”

Here is the objection I take most seriously, because the people making it are right about a lot. The prevention purist (the SLSA-and-pin-your-actions camp) says: supply-chain security is an architecture problem. Harden the build, prove provenance, pin everything, and stop pretending a SOC queue is going to catch a nation-state that spent two years earning maintainer trust. Detection here is theater; go fix the pipeline.

I agree with the priority. I do not agree that it is sufficient, and the reason is that prevention has three structural gaps the incident record keeps walking through.

Gap one: inherited trust defeats pinning. The tj-actions compromise did not start at tj-actions. The attacker stole a Personal Access Token from the SpotBugs project’s workflow, moved laterally to a reviewdog maintainer’s token (CVE-2025-30154), and used that to push malicious code into tj-actions, whose original target was Coinbase, reached entirely through its build dependencies [7]. Pinning your direct dependency to a SHA does nothing when the compromise arrives transitively, through the trust your pinned dependency itself extends to its dependencies. You cannot SHA-pin the entire transitive closure of everyone’s tokens. This is the living-off-the-land problem relocated to the build: the danger is not one bad binary, it is the chain of individually-trusted steps: and the runner is the EDR-blind layer where that chain executes.

Gap two: ephemeral runners still have an exfiltration window. “Destroy the runner after every job” bounds persistence, not theft. Shai-Hulud did its damage during a legitimate npm install inside an ordinary job: harvest secrets, exfiltrate, done, all inside the runner’s normal lifetime [3]. The runner being ephemeral did not matter, because the attack completed before the runner was ever torn down. Ephemerality is a real control against persistence and no control at all against a smash-and-grab that finishes in the seconds it takes to run one build step.

Gap three: provenance proves the “how,” not the “whether it was malicious.” SLSA attests that an artifact was built on a specific platform, from specific inputs, by a specific process. It does not attest that the process was benign. A malicious step that runs inside your L3-compliant, fully-attested pipeline produces an artifact with perfectly valid provenance. Provenance answers “was this built the way we said,” not “did something evil run while we built it.” Those are different questions, and only the second one is a detection question.

Prevention is necessary. It is also, on the evidence, structurally incomplete, and the incompleteness is precisely the behavioral, in-flight surface that only telemetry sees.

What to Actually Detect on the Build

Three telemetry sources cover the surface prevention cannot. I am going to be honest about where each one works and where you have nothing, because pretending otherwise is how you end up with green coverage over a blind spot.

1. Runner process and network telemetry. This is the crown jewel and the hardest to get. On a self-hosted runner you control the host, so you can run an eBPF-based collector (Falco or Tetragon are the common modern choices) and see the full process tree: Runner.Worker spawning a shell spawning a network tool, plus the outbound connections each step makes. That is exactly the resolution you need, because the malicious step is almost always a descendant of the legitimate runner agent reaching somewhere it should not. Depth of visibility is a cost-and-effort tradeoff; a raw execve-and-connect feed is cheap, full syscall context is not, and you should size that to how much you trust your build inputs.

The honest limitation: GitHub-hosted runners give you none of this. You do not own the host, you cannot install an agent, and you get no process or network telemetry from inside the job. If your builds run on hosted runners, this detection is simply unavailable, and the only responsible thing is to say so rather than paper over it. You compensate one layer out, with OIDC and CloudTrail, which is the next source.

2. OIDC token exchanges and downstream cloud actions. Even when the runner internals are dark, the consequences of a compromised build show up in the cloud audit log. When a GitHub Actions job assumes an AWS role via OIDC, CloudTrail records an AssumeRoleWithWebIdentity call, and the token carries claims identifying the repo, branch, and workflow that requested it [8]. That is a behavioral fingerprint of your build’s identity. A deploy that suddenly assumes a role from an unexpected branch, or a sub claim that does not match the identity that normally deploys that service, is visible in CloudTrail regardless of whether you can see inside the runner. This is the workload-identity telemetry from Workload Identities Have a Home Address applied at the build boundary.

3. Pipeline-configuration drift. The TensorFlow self-hosted runner compromise Praetorian documented worked because a fork pull request could modify the workflow’s runs-on field and land itself on a privileged self-hosted runner that public forks were never meant to touch [9]. A workflow file that changes which runner it targets, grants itself new permissions, or adds a step that reads a secret it never read before is a config-drift signal you can compute from the repo itself, no runtime telemetry required. Diff the effective pipeline config against its baseline and alert on privilege-relevant changes.

One Rule: Runner Egress During a Secret-Bearing Job

Here is a detection for the highest-value case, the one Shai-Hulud and tj-actions both are: a build step that touched secrets and then reached out to a destination you never approved. It is deliberately not “alert on any outbound connection,” because builds pull from registries and CDNs constantly. It fires only when the chain closes: the runner agent’s own process ancestry leads to a network tool, that tool connects to a non-allowlisted host, and it happens inside a job window where secrets were exposed.

It runs on self-hosted-runner eBPF telemetry (Falco/Tetragon-style execve + connect records). On GitHub-hosted runners you will not have this feed; fall back to the OIDC/CloudTrail sidebar below.

# Detection rationale: a poisoned build step (tj-actions, Shai-Hulud) reads secrets
# out of the runner and exfiltrates them to attacker infrastructure, all inside a
# legitimate job. No single event is malicious: builds spawn shells, shells run curl,
# curl connects to hosts, jobs read secrets. The ATTACK is the SEQUENCE: a process
# descended from Runner.Worker opens an outbound connection to a NON-allowlisted
# destination during a window where secret material was accessed. Keyed on the
# runner-agent ancestry (rename-resistant: it does not matter what the exfil tool
# is called, only that Runner.Worker is its ancestor) plus a network egress the
# build's own dependency mirrors and registries were allowlisted to exclude.
# Requires host-level telemetry: self-hosted runners only (eBPF: Falco/Tetragon).
# GitHub-hosted runners give no such feed; compensate with the CloudTrail sidebar.
title: Self-Hosted CI Runner Secret-Bearing Job Egress to Non-Allowlisted Host
status: experimental
logsource:
  product: linux
  category: network_connection   # eBPF connect() records from a self-hosted runner
detection:
  # Leg 1: the connecting process descends from the runner agent, not from the OS.
  runner_ancestry:
    ParentImage|contains:
      - 'Runner.Worker'          # GitHub Actions runner worker
      - 'runner.worker'
      - '/actions-runner/'       # self-hosted runner install path
    Image|endswith:
      - '/curl'
      - '/wget'
      - '/nc'
      - '/python3'
      - '/node'
      - '/bash'
      - '/sh'
  # Leg 2: egress leaves the approved build-dependency perimeter.
  approved_egress:
    DestinationHostname|contains:
      - 'registry.npmjs.org'
      - '.pkg.github.com'
      - 'pypi.org'
      - 'files.pythonhosted.org'
      - '.blob.core.windows.net'    # your artifact / mirror hosts, tune per env
  # Leg 3: this job accessed secret material (env-var or mounted secret read),
  # correlated per runner+job so egress alone on a clean job stays quiet.
  secret_access:
    FileName|contains:
      - '/run/secrets/'
      - '.git-credentials'
      - '/.npmrc'
      - '/.docker/config.json'
  condition: runner_ancestry and secret_access and not approved_egress
  correlation_scope:
    group_by: [ 'runner_id', 'ci_job_id' ]
    timespan: 15m
falsepositives:
  - A new legitimate registry, CDN, or package mirror not yet in the allowlist:
    the first build after adding it fires once; add the host and re-baseline.
  - First run after a dependency is added that pulls from a new upstream host.
  - Self-hosted internal mirrors reached by IP rather than hostname: allowlist by
    CIDR as well as name.
  - Coverage/test uploaders (Codecov successors) that legitimately post outside the
    registry perimeter: allowlist the specific uploader destinations explicitly.
level: high
tags:
  - attack.exfiltration
  - attack.t1041          # Exfiltration Over C2 Channel
  - attack.credential_access
  - attack.t1552.001      # Credentials In Files
  - attack.t1552.007      # Container/CI credentials via API/env

The rule is a per-job, per-runner correlation across three individually-legitimate events. That is the same shape as every chain detection worth writing: the signal is in the relationship, not any single leg. And it is honest about its blind spot in the header: no host telemetry, no rule.

When you cannot see inside the runner, watch the cloud it talks to. Every OIDC-based deploy leaves an AssumeRoleWithWebIdentity in CloudTrail whose token claims name the repo and branch that requested it [8]. Baseline which sub claim normally deploys each production role, then alert when a privileged role is assumed by a claim that does not match: a deploy role assumed from a feature branch, a fork, or a repo that has no business touching that service. It is a weaker signal than runner telemetry, because it only fires once the compromised build reaches for cloud access, but it is the one detection that works even on fully-hosted runners you cannot instrument. Treat it as the compensating control for the visibility you do not have, not as an equal substitute for the runner feed you should be trying to get.

What to Build This Quarter

Stop treating CI as plumbing. It is a production tier that runs untrusted code with credentials to everything you ship, and the four incidents that defined supply-chain risk all lived there, not in the code you reviewed. Scanning will not see the next one, because the next one will run as a legitimate step inside a legitimately-versioned dependency and exfiltrate before your SBOM is even generated.

Harden first (pin to SHAs, make runners ephemeral, scope tokens with OIDC, attest with SLSA) and let that hardening manufacture your signals. Then build these three, in this order:

OIDC claim-mismatch detection in CloudTrail. It works everywhere, including hosted runners, and it catches the compromised build the moment it reaches for cloud access. Start here because it has the fewest prerequisites.
Runner egress-during-secret-access correlation on your self-hosted fleet, via eBPF. This is the one that would have caught tj-actions and Shai-Hulud in the act. It costs the most to stand up and it is worth it, but only where you own the host.
Pipeline-config drift detection on privilege-relevant workflow changes: runs-on, permissions, new secret reads. Cheapest of the three, computed straight from the repo, and it closes the fork-PR runner-hijack path.

And say the quiet part out loud in your coverage report: on GitHub-hosted runners, detections one and three are all you have, because the runner internals are not yours to see. That is not a gap to hide behind a green dashboard. It is the honest boundary of what the build will let you watch, and knowing exactly where it falls is the difference between coverage and comfort.

Resources

The XZ Backdoor (CVE-2024-3094). OpenSSF, March 2024 (CVSS 10.0; malicious payload injected during the liblzma build process, absent from the reviewed source tree). See also CISA alert.
GHSA-mrrh-fwg8-r2c3: tj-actions/changed-files (CVE-2025-30066). GitHub Advisory / CISA alert, March 2025 (~23,000 repos; tags retargeted to a malicious commit that dumped runner memory secrets into public workflow logs).
Widespread Supply Chain Compromise Impacting the npm Ecosystem (Shai-Hulud). CISA, September 2025 (self-replicating worm harvested secrets with TruffleHog during install and exfiltrated to attacker GitHub repos). Analysis: Wiz.
Analysis of the Codecov Supply Chain Compromise. Rapid7, April 2021 (modified Bash Uploader exfiltrated CI environment variables from ~23,000 customers).
OWASP Top 10 CI/CD Security Risks. OWASP (CICD-SEC-4 Poisoned Pipeline Execution; CICD-SEC-10 Insufficient Logging and Visibility).
SLSA Build Levels. SLSA v1.0 (L3 requires a hardened platform isolating runs and protecting signing material; provenance attests how an artifact was built, not that the build was benign).
SpotBugs Access Token Theft Identified as Root Cause of GitHub Supply Chain Attack. The Hacker News, April 2025 (stolen SpotBugs PAT → reviewdog CVE-2025-30154 → tj-actions; original target Coinbase). See also Unit 42 threat assessment.
OpenID Connect: GitHub Actions. GitHub Docs (OIDC token carries repo/branch/workflow claims; AWS AssumeRoleWithWebIdentity is recorded in CloudTrail with the assumed-role identity).
TensorFlow Supply Chain Compromise via Self-Hosted Runner Attack. Praetorian, January 2024 (fork PR modifies runs-on to land on a privileged self-hosted runner and steal a PAT).