Pin every package in your Dockerfile, or a routine apt-get install will bite you later

We run CosmicIRC, a small independent IRC network, on a handful of eggdrop bots in Docker. One of them, Pulsar, went from "stable for months" to "dead every few minutes" the moment we added two packages to its Dockerfile. Neither package was the bug. That's the part worth writing down.

The change that shouldn't have mattered

We were adding a typo-tracking feature to Pulsar (that project has its own crash story — see the crash that survived every fix) and it needed a spell-checker. So we added aspell and aspell-en to the Dockerfile's single apt-get install line, alongside eggdrop, tcllib, tcl-tls, ca-certificates, and openssl — all packages that had been sitting there, unpinned, working fine, for a long time.

Rebuilt. Deployed. Pulsar started dying with exitCode=139 — SIGSEGV — anywhere from 7 to 284 seconds after boot. No pattern to the interval, no stack trace, no line in the eggdrop log pointing at anything. It just stopped existing and docker events recorded a signal.

The instinct here is to suspect the thing you just changed. We suspected aspell. It shells out to a subprocess, it does pipe I/O, it felt like the kind of thing that could destabilize a Tcl event loop. That instinct was wrong, and chasing it would have wasted a lot of time — aspell hadn't even been wired into the bot's code yet. It was installed but unused. The crash was happening before any of the new feature's code ran.

What actually happened

The Dockerfile's apt-get install line had zero version pins. Every package name, no version number. That's a common shortcut — it's less to maintain, and it works right up until it doesn't.

Docker layer caching means that line only re-runs apt-get update when the line itself changes. Ours had been stable for a long time, so the image had been rebuilding from a cached layer with whatever package versions were resolved the first time that line ever ran. Adding aspell aspell-en to the end of the line changed the line's text, which invalidated the cache, which forced a fresh apt-get update against the current package repositories — and pulled current versions of every package on that line, not just the two new ones.

That silently upgraded tcl-tls, openssl, ca-certificates, and eggdrop itself to whatever was current on the day of the rebuild, with no changelog review, no testing, and no indication in the build output that anything meaningfully different had happened. One of those combinations — almost certainly something in tcl-tls's TLS handshake path interacting with the newer OpenSSL — segfaulted under real use.

We confirmed this by comparing docker events --filter container=pulsar output across builds: the crash-looping build was the one with the fresh apt-get update layer; rolling back to the last cache-hit build (before the Dockerfile edit) was stable. That isolated the cause to "something changed in the dependency graph," which pointed straight at the unpinned install line rather than at aspell.

The fix

Pin every package on that line to the exact version that was already known to work, not just the new ones you're adding:

eggdrop=1.8.4+repack1-0.1build2
tcllib=1.21+dfsg-1
tcl-tls=1.7.22-3build2
ca-certificates=20260601~24.04.1
openssl=3.0.13-0ubuntu3.15
aspell=0.60.8.1-1build1
aspell-en=2020.12.07-0-1

With every version explicit, editing the line to add or remove a package no longer has any effect on the versions of the packages that were already there. A cache invalidation just reinstalls the same versions faster — it can't silently substitute anything.

The lesson, generalized

An unpinned apt-get install line isn't broken while it's untouched. It's a landmine that looks exactly like ordinary, working infrastructure, and the person who steps on it is usually making an unrelated change — adding a package, reordering a line, anything that touches that layer's cache key. The bug report they file will be about the thing they just added, because that's the only thing that changed from their point of view. The real cause is everything they didn't touch.

If you maintain a Dockerfile with an install line you haven't looked at in months: check whether it's pinned. If it isn't, pin it now, while nothing is on fire, rather than while you're staring at exitCode=139 and a log file with nothing in it.

We run CosmicIRC, a free, independent IRC network. If you've chased a "the new thing I added broke something old" bug back to an unpinned dependency, we'd like to compare notes — we're at irc.cosmicirc.com, #cosmic.