"GitHub's own security guidance recommends pinning actions to full commit SHAs as the only truly immutable way to consume an action"
Why doesn't GitHub just enforce immutable versioning for actions? If you don't want immutable releases, you don't get to publish an Action. They could decide to enforce this and mitigate this class of issue.
> Why doesn't GitHub just enforce immutable versioning for actions?
I always wish these arguments came with a requirement to include a response to "well, what about the other side of the coin?", otherwise, you've now forced me to ask: well?
The two sides of the coin: Security wants pinned versions, like you have, so that compromises aren't pulled in. Security does not want¹ pinned versions, so that security updates are pulled in.
The trick, of course, is some solution that allows the latter without the former, that doesn't just destroy dev productivity. And remember, …there is no evil bit.
(… I need to name this Law. "The Paradox of Pinning"?)
(¹it might not be so explicitly state, but a desire to have constant updated-ness w/ security patches amounts to an argument against pinning.)
Their question isn't about pinned versions, it's about immutable versions. The question is why it is possible to change what commit "v5" refers to, not "why would you want to write v5".
You already don't get updates pulled in with the system unless they swap the version out from under you, which is not a normal way to deploy.
It amounts to an argument against pinning in a (IMO) weird world view where the package maintainer is responsible for the security of users' systems. That feels wrong. The user should be responsible for the security of their system, and for setting their own update policy. I don't want a volunteer making decisions about when I get updates on my machine, and I'm pretty security minded. Sure, make the update available, but I'll decide when to actually install it.
In a more broad sense I think computing needs to move away from these centralised models where 'random person in Nebraska'[0] is silently doing a bunch of work for everyone, even with good intentions. Decisions should be deferred to the user as much as possible.
I assume this is because it is modeled after git tags, and at this point it would be a major change to move away from this. But it should probably get started at some point.
My initial thought is that if this isn't a new compromise, Trivy must not have rotated the old credentials. They claim, however,
> We rotated secrets and tokens, but the process wasn't atomic and attackers may have been privy to refreshed tokens
… does anyone know what exactly they're talking about, here? To my knowledge, GH does not divulge new tokens after they're issued, but it depends on the exact auth type we're talking about, and GH has an absurd number of different types of tokens/keys one can use.
OpenClaw creator made some related claims, that as soon as he created a GitHub organization with a new name, somehow it was stolen from him, and he had to ask Github people to do it for him atomically.
So the first incident was on March 19th and the second incident is March 22nd —- evidently the attackers maintained persistence through maybe two separate credential rotation efforts.
Well, not my best 2 weeks at work, now I have to fill out a dozen forms and sit trough a shitload of meeting, just because they got pwned (twice, or once, but really badly :D )
Friendly reminder that just because someone is building security software it doesn't mean they are competent and won't cause more harm than good.
Every month the security team wants me to give full code or cloud access to some new scanner they want to trial. They love the fancy dashboards and lengthy reports but if I allowed just 10% of what they wanted we would be pwned on the regular...
I audited Trivy's GitHub Actions a while back and found some worrying things, the most worrying bit was in the setup-trivy Action where it was doing a clone of main of the trivy repo and executing a shell script in there. There was no ref pinning until somebody raised a PR a few months ago. So a security company gave themselves arbitrary code execution in everyone's CI workflows.
Aqua were breached earlier this month, failed to contain it, got breached again last week, failed to contain it again, and now the attackers have breached their Docker Hub account. Shit happens but they're clearly not capable of handling this and should be enlisting outside help.
Granting broad access to "security" tools so some vendor can take another shot at your prod keys is not risk reduction.
Most of these things are just report printers that makes more noise than a legacy SIEM, and once an attacker is inside they don't do much besides dump findings into a dashboard nobody will read.
If you want less self-inflicted damage, stick new scanners in a tight sandbox, feed them read-only miror data, and keep them away from prod perms until they have earned trust with a boring review of exactly what they touch and where the data goes.
Otherwise you may as well wire your secrets to a public pastebin and call it testing.
Yet many of these tools have setup like: create a service account, give it about thousand permissions (if not outright full ownership) and send us the JSON private key.
My hypothesis is that generally, there's no quality floor at which security departments are "allowed" to say "actually, none of the options on the market in this category are good enough; we're not going to use any of them". The norm is to reflexively accept extreme invasiveness and always say yes to adding more software to the pile. When these norms run deeply enough in a department, it's effectively institutionally incapable of avoiding shitty security software.
Fwiw w/r/t Trivy in particular,I don't think Trivy is bad software and I use it at work. We're unaffected by this breach because we use Nix to provide our code scanning tools and we write our own Actions workflows. Our Trivy version is pinned by Nix and periodically updated manually, so we've skipped these bad releases.
Most of corporate security nowadays involves "endpoint security solutions" installed on all devices, servers and VMs, piping everything into an AI-powered dashboard so we can move fast and break everything.
Wasn't this discovered already last week, on Friday, that the threat actor had replaced the legit images with malware images? And republished 75 out of 76 tags?
No, the actor reappeared. This article is not fully updated. On March 22nd, the actor compromised their DockerHub account and published new Docker images.
The sandbox will need internet access (to update data) and you will need to send code to test into it; so compromise already equals leaking all your code, without even breaking the sandboxing
> The sandbox will need internet access (to update data) and you will need to send code to test into it; so compromise already equals leaking all your code, without even breaking the sandboxing
Compromising all code in one directory is bad.
Compromising all my data in all other directories, including mounted cloud drives, is worse.
I restrict most dev tools to access only the current directory.
Whenever possible, credentials shouldn't be inside the sandbox either. Credential proxying, or transparent credential injection, for example with Sandcat: https://github.com/VirtusLab/sandcat
second breach in a month from the same initial credential compromise. the first rotation didn't fully revoke access. the attacker walked right back in. no persistence needed.
Why doesn't GitHub just enforce immutable versioning for actions? If you don't want immutable releases, you don't get to publish an Action. They could decide to enforce this and mitigate this class of issue.
I always wish these arguments came with a requirement to include a response to "well, what about the other side of the coin?", otherwise, you've now forced me to ask: well?
The two sides of the coin: Security wants pinned versions, like you have, so that compromises aren't pulled in. Security does not want¹ pinned versions, so that security updates are pulled in.
The trick, of course, is some solution that allows the latter without the former, that doesn't just destroy dev productivity. And remember, …there is no evil bit.
(… I need to name this Law. "The Paradox of Pinning"?)
(¹it might not be so explicitly state, but a desire to have constant updated-ness w/ security patches amounts to an argument against pinning.)
You already don't get updates pulled in with the system unless they swap the version out from under you, which is not a normal way to deploy.
In a more broad sense I think computing needs to move away from these centralised models where 'random person in Nebraska'[0] is silently doing a bunch of work for everyone, even with good intentions. Decisions should be deferred to the user as much as possible.
[0]: https://xkcd.com/2347/
Allowing it to be updated can also fix security problems.
It’s basically all the same arguments as static vs dynamic linking.
Plus, I believe I saw that the one action was getting the latest version of trivy anyway.
> We rotated secrets and tokens, but the process wasn't atomic and attackers may have been privy to refreshed tokens
… does anyone know what exactly they're talking about, here? To my knowledge, GH does not divulge new tokens after they're issued, but it depends on the exact auth type we're talking about, and GH has an absurd number of different types of tokens/keys one can use.
Trivy ecosystem supply chain temporarily compromised - https://news.ycombinator.com/item?id=47450142 - March 2026 (35 comments)
So the first incident was on March 19th and the second incident is March 22nd —- evidently the attackers maintained persistence through maybe two separate credential rotation efforts.
Every month the security team wants me to give full code or cloud access to some new scanner they want to trial. They love the fancy dashboards and lengthy reports but if I allowed just 10% of what they wanted we would be pwned on the regular...
Aqua were breached earlier this month, failed to contain it, got breached again last week, failed to contain it again, and now the attackers have breached their Docker Hub account. Shit happens but they're clearly not capable of handling this and should be enlisting outside help.
If you want less self-inflicted damage, stick new scanners in a tight sandbox, feed them read-only miror data, and keep them away from prod perms until they have earned trust with a boring review of exactly what they touch and where the data goes. Otherwise you may as well wire your secrets to a public pastebin and call it testing.
Yet many of these tools have setup like: create a service account, give it about thousand permissions (if not outright full ownership) and send us the JSON private key.
At least they make the red flag nice and obvious.
Fwiw w/r/t Trivy in particular,I don't think Trivy is bad software and I use it at work. We're unaffected by this breach because we use Nix to provide our code scanning tools and we write our own Actions workflows. Our Trivy version is pinned by Nix and periodically updated manually, so we've skipped these bad releases.
Compromising all code in one directory is bad. Compromising all my data in all other directories, including mounted cloud drives, is worse.
I restrict most dev tools to access only the current directory.
So, stealing credentials in the current directory and in all other directories are the same thing?