Hashes and Signatures

Hash proves integrity, signature provides provenance

security by Zsolt Bizderi

Contents

Hashes

A cryptographic hash function takes an input of any size and produces a fixed-length output called a digest. SHA-256 results in 256 bits. The same input always produces the same digest, changing a single bit in the input changes about half of the output, and although collisions do exist, you cannot feasibly work backwards from a digest or find two inputs that produce the same one.

This makes hashes very good at detecting change. When you run sha256sum -c against a checksum file and get OK, the file on your disk is byte-for-byte identical to the file the checksum was generated from. That flags corrupted downloads, truncated transfers, bit rot on a dying disk, and a mirror serving an older version. The problem is that a hash has no key, therefore anyone can generate one, including an attacker. If the checksum is published on the same server as the download and that server gets compromised, the attacker replaces both files and the hash check still passes. You have confirmed the file matches the checksum, but the attacker wrote both of them.

So a hash gives you integrity in the narrow sense, meaning the bytes have not been altered since the digest was calculated. It has no idea who calculated it.

HMAC

HMAC mixes a secret key into the hash, so only someone holding the key can produce a valid tag. You see this in API request signing, webhook verification and session tokens.

HMAC is symmetric though. Both ends hold the same secret, so either side could have generated any given tag. You have authentication between two parties who already trust each other, but you do not get non-repudiation. It also doesn't scale to software distribution, because you can't give the same secret to thousands of strangers and expect it to stay secret.

Signatures

A digital signature uses an asymmetric key pair. The publisher hashes the file and signs the digest with their private key, and anyone with the matching public key can verify that the signature was made by that private key over those exact bytes.

The hash is still doing the integrity work underneath. The signature ties that digest to a key, so if the file changes the signature breaks and producing a new valid signature needs the private key. An attacker who owns the web server can swap the tarball, but they cannot forge a signature that verifies against the publisher's public key. This is provenance, and because only the key holder could have produced the signature, you also get non-repudiation.

All three side by side:

import hashlib, hmac
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey

release  = b"xz-5.6.1 tarball"
tampered = b"xz-5.6.1 tarball + backdoor"

# 1. Plain hash: anyone can compute it
print(hashlib.sha256(release).hexdigest())
print(hashlib.sha256(tampered).hexdigest())   #different, but an attacker can publish this one too

# 2. HMAC: needs the shared secret
secret = b"shared-between-both-ends"
tag = hmac.new(secret, release, hashlib.sha256).digest()
check = hmac.new(secret, release, hashlib.sha256).digest()
print(hmac.compare_digest(tag, check))        #true, but either end could have made it

# 3. Signature: only the private key signs, anyone with the public key verifies
maintainer = Ed25519PrivateKey.generate()
public_key = maintainer.public_key()
sig = maintainer.sign(release)

public_key.verify(sig, release)               #no exception, valid
try:
    public_key.verify(sig, tampered)
except InvalidSignature:
    print("tampered file rejected")

So while every check in there is about bytes and keys, none of them help detect malicious behaviour.

Where does the public key come from? If you pull the key from the same server as the tarball, you are back to the checksum problem with extra steps. The key needs to come from somewhere independent, like a distro keyring that came with your OS or a fingerprint confirmed through a separate channel, or a key that has signed the previous releases. This is why PKI and the web of trust exist.

What does provenance cover? A valid signature proves the file was signed by whoever controls that private key. It cannot determine whether that person is honest, whether their laptop was compromised, or whether the code matches the changelog. It shows you who to blame afterwards and that's about it.

The xz-utils backdoor (CVE-2024-3094)

xz-utils provides liblzma, a compression library that ends up linked into a huge part of most Linux systems. For years it was maintained by one person, Lasse Collin, in his spare time.

The attack chain looked like this:

  1. Build trust: In 2021 an account called Jia Tan started contributing patches to the project.
  2. Apply pressure: Over the next year accounts turned up on the mailing list complaining about slow releases and pushing Collin to take on a co-maintainer. Collin had been open about his limited capacity and gave Jia Tan more and more access.
  3. Take over releases: By 2023 Jia Tan was cutting releases and signing them with their own key.
  4. Hide the payload: The malicious code was stored in two binary test files, bad-3-corrupt_lzma2.xz and good-large_compressed.lzma and committed to the git repo.
  5. Hide the trigger: Autotools projects ship release tarballs with generated build scripts that are not tracked in git. The 5.6.0 and 5.6.1 tarballs from February and March 2024 included a modified build-to-host.m4 that did not exist in the repo. During ./configure it pulled the payload out of the test files and injected a precompiled object into liblzma, but only on x86-64 Linux with glibc when building a Debian or RPM package. Building from a git checkout gave you a clean library.
  6. Hit sshd: OpenSSH does not link liblzma directly, but several distros patch sshd to notify systemd on startup, which pulls in libsystemd, which pulls in liblzma. The injected code used glibc's IFUNC mechanism to hook RSA_public_decrypt. When a client connected with a certificate carrying a command signed by the attacker's Ed448 key, sshd ran it as root before authentication.

It made it into Debian unstable and testing, Fedora Rawhide and 41, openSUSE Tumbleweed and Kali.

Andres Freund, a PostgreSQL developer at Microsoft noticed SSH logins on his Debian sid box were taking about half a second longer than they should, followed that through some valgrind errors, and posted it to oss-security on 29 March 2024.

It got a CVSS score of 10.0.

Running both checks against it

If you were a distro packager pulling in 5.6.1, this is approximately what you would have seen:

$ sha256sum -c SHA256SUMS
xz-5.6.1.tar.gz: OK         #integrity check successful

$ gpg --verify xz-5.6.1.tar.gz.sig xz-5.6.1.tar.gz
gpg: Good signature from "Jia Tan"   #provenance: signed by the actual maintainer

Both results are correct.

The tarball was delivered untouched and it came from exactly who it said it came from. The attacker spent two years getting into a position where their signature was legit and verifiable, so the controls did their job and the backdoor went straight through.

The backdoor even used a signature for its own protection. Every command had to be signed with the attacker's private Ed448 key, with the public half embedded in the implant, so nobody else could use it. Researchers could fully reverse engineer the thing and still could not trigger it. That is provenance working exactly as designed, just for the attacker.

Controls that would have detected it:

  • Diffing the tarball against the tagged source would have shown the modified build-to-host.m4, as it was the only file in the release that did not exist in git. If you build your own tarball from the tag and hash both, the digests do not match. This is a hash doing its job properly, because you now have two independently produced files to compare.
  • Build provenance from SLSA or Sigstore attests to where an artefact was built, from which commit and on which builder. A tarball built on a maintainer's own machine and uploaded by hand has nothing to attest to.
  • Build scripts and binary test data should be reviewed like code.
  • The people side: A new maintainer taking over after a pressure campaign from accounts with no footprint anywhere else is a supply chain signal.

Smuggling and stego

This is adjacent to ASCII Smuggling and Steganography in the Wild. Tag characters are valid Unicode, LSB-encoded pixels still produce a perfectly valid PNG, and the XZ test files were the sort of broken archives you would expect to find in a compression test suite. In all of these cases, the format is valid, the hash matches and the signature checks out. HalluSquatting is the dependency side of the same problem, where the package is intact and signed.

The hash and signature can confirm that the content is authentic and unchanged, but they cannot tell you whether the content itself is what you actually intended to receive.

What can you do about it?

This comes down to attention to detail and verification controls.

  • Don't trust a checksum hosted next to the download, as it protects against corruption and nothing else. Pin dependency hashes in lockfiles (package-lock.json, go.sum, requirements.txt with --require-hashes) so a changed artefact under the same version number makes the build fail.
  • Verify signatures against a key you got independently. Save the fingerprint the first time you trust a project and alert when the signing key changes, because a new key on a routine release should raise red flags.
  • Prefer artefacts with build provenance. Sigstore signs with short-lived certificates tied to an OIDC identity and logs every signature in the Rekor transparency log, so a release signed outside the normal pipeline stands out. Check SLSA attestations if projects publish them.
  • Build from source tags for anything critical and compare against upstream. Reproducible builds let you compare your hash against upstream's, which makes the hash a meaningful control again.
  • Keep an SBOM. When CVE-2024-3094 dropped, everyone's first question was whether 5.6.0 or 5.6.1 was running anywhere in their estate. With an SBOM you can run a query in minutes.
  • If you publish software yourself, treat the signing key as a critical asset. Sign in CI with hardware-backed or short-lived keys, require two approvals on release workflows, and treat any change to who can sign as a change control event.

Lastly, find a way to support the open source projects you depend on. xz was one volunteer away from compromise.

Hash your downloads and verify your signatures, just remember a green OK and a good signature only tell you that nothing changed and who signed it :)

Related