Session 05

The miner that never ran

Upgrading our production Project-Ops frontend, we found a crypto-miner dropper and an SSH key nobody on our side put there. Neither ever ran. But mapping what that foothold could reach turned up a real leak…

Commits
7
Systems
2
Read
9min
Product capture

Title card, The miner that never ran, beside a printed incident receipt: two planted files, zero times either ran, one real leak found and contained in 12 minutes, logs cover about 3 percent of the window, evidence of data access none found, proof of no access not possible.

Upgrading our production Project-Ops frontend, we found a crypto-miner dropper and an SSH key nobody on our side put there. Neither ever ran. But mapping what that foothold could reach turned up a real leak of our own.

Two files nobody put there

The upgrade on Sunday was routine. Project-Ops is the project and task app in our Unicorn Commander suite, and Persephone, my main assistant, was moving its production frontend onto a patched build. Production had drifted from git: its image dated from August, an older commit plus a production-only patch and a hand-edited config file. So before replacing the running container, she diffed it against git. The diff showed the drift we expected, and two files we didn't.

The diff showed the drift we expected, and two files we didn't.

One was a small Python script that fetches and starts XMRig, the open-source crypto-miner, with a file time of September 13. The other, with a file time of September 4, was an SSH key with a generic placeholder comment, in a file called authorized_keys2. That is a second keys file that SSH still reads by default and most people never look at.

The receipt: real output, filtered, with our container, directory and user names replaced by placeholders.

The way in was React2Shell, CVE-2025-55182, the remote-code-execution flaw in React Server Components that reached Next.js apps. Our frontend ran Next.js 15.5.4. The fix had been public since December 3, 2025, nine months before the first planted file. That one is on us. The production image, built in August, still carried a version from before the fix, and it sat there until this upgrade. Public reports describe automated campaigns exploiting React2Shell within days of disclosure, many of them dropping XMRig. A miner plus a generic key fits that pattern. Nothing we found suggests it was aimed at us.

Why it never ran

The dropper is a Python script, and the image has no Python. The key is for SSH, and the container has no SSH server. Neither server showed a miner process or a connection to a mining pool, and the diff showed no miner binary on disk. No other container on either of our main servers had the files. The key was in no authorized-keys file we could read on the ten machines we checked.

At 1:52 p.m. the evidence was saved. At 2:04 p.m. the patched build was live: Next.js 15.5.28, React 19.1.10, and an app directory owned by root. The old Dockerfile had handed that whole directory to the user the app runs as, which is how a remote-code bug got to drop a file next to the app. Now only the build cache is writable. The compromised image went into evidence, untouched.

The new build did break sign-in for nine minutes. A race in the sign-in callback logged people out whenever the profile request took longer than 50 milliseconds. Our test ring sits on the local network, where that never happens. Production is on the internet, where it happened every time. Fixed at 2:13.

Then we swept every other Next.js app on both servers. One other app was still on a version open to this exact flaw. Within the hour it had a block at the edge for the requests the exploit uses, and its patched build shipped that night. Six more were rebuilt overnight on patched versions, one at a time, each behind a temporary block that came down only after its checks passed. The queue finished at 1:48 a.m.

Fifty-one days unnoticed. Twenty-seven hours to close it out.

What the foothold could reach

A payload that never runs can still mean someone had a look around. So on Monday we mapped what that position could reach without credentials, from throwaway containers on the same two networks it sat on: no tokens, no cookies, status codes only.

Everything that holds data said no. The database wanted a password. So did the cache. The file store returned 403, and every business route returned 401: projects, tasks, users, customers, contacts, meetings. The container itself held no secrets, only public settings, and nothing was mounted into it.

Code execution, then a lot of locked doors. And one that wasn't.

Then the probe found one door that was open, and it had nothing to do with the intruder. The public status endpoints of Meeting-Ops reported the health of the services behind them, including the full address of our speaker-recognition service with its password in it. No login needed, from anywhere on the internet. The code that echoed it dated from May. The password had been in it since late September at the latest. Anyone who looked could have read it, intruder or not. What that password guarded was compute, not data. The service runs our speaker models on audio the caller sends. It keeps none of it and can reach no recordings, so the password let someone run our models on their own audio, not read anything of ours or anyone else's. It does not log successful requests, so we cannot say whether anyone did.

We confirmed it from outside at 1:52 p.m. By 2:04 p.m. the voice service answered only an allowlist of our own addresses. Twelve minutes, at the same minutes on the clock as the day before. A coincidence, not a process.

The real fix went live at 4:58 p.m. A coding agent wrote it in a sandbox and then timed out babysitting the test suite. Persephone reviewed its work, repaired its worktree and finished it. Every URL the status endpoints return, and the logs, now show the credential as ***. The full suite hangs when run in one go, so each of its 166 files ran on its own, and all 1,606 tests passed. The runbook said to restart one service; she checked and found two workers carrying the same password, so all three moved together. Then she rotated the password on my go. The voice service answered 200 with the new one and 401 without. The allowlist stays as a second lock.

A health check that said too much. Placeholders in angle brackets; no real value appears.

What we cannot prove

Here is the honest limit. From the first planted file to the replacement was about 51 days. The logs that record who connected and reach back into that window cover about 41 hours of it, roughly 3 percent. Both show zero requests from the compromised container. Most app logs start after the container was replaced, because our own redeploys between October 24 and 26 deleted the older ones, and several never record who connected at all. The edge proxy's log covers the whole window but keeps only errors, and traffic between containers never passes through it. It shows no failed request from that network.

What our logs could see.

One path we cannot rule out. A foothold inside a web server sees what passes through it, and our sign-in flow handed each new session its token in the URL's query string, which the server receives. We have no evidence that anything read those tokens, and we can't prove nothing did. The fix moves the token after the # in the URL, which browsers never send to the server, and the page wipes it from the address bar on arrival. Both halves are live, and every session token issued before the fix has been revoked.

So the sentence we can stand behind is "no evidence of data access." Not "no data was accessed." The logs don't reach back far enough to say that, so we won't.

What we changed

We diff before every promote. docker diff lists every file a running container has added or changed since it was created from its image, so files like these show up directly. Running it is now a standing rule before any container is promoted or recreated.

The app can't rewrite its own code. The Project-Ops frontend's directory now belongs to root. Only the build cache stays writable.

Production builds from main. The Project-Ops frontend's production-only patch and hand-edited file are gone. Per-environment settings now go in at build time, so there is no private branch left to drift.

Tokens stay out of query strings. Both halves of the fix are live, and tokens issued before it no longer work.

Status endpoints don't echo secrets. Meeting-Ops now masks them.

Evidence gets pinned on purpose. Our nightly cleanup deletes any image unused for a day. It would have deleted the compromised image on Monday night. Persephone caught it, pinned it, and copied it off the server with a checksum.

Two things caught this, and neither was an alert. One was a habit: compare what is running with what is in git before you touch it. The other was an assistant that checks instead of assuming. Persephone could have rebuilt production from main and moved on, and the planted files would have vanished with the old container, unseen. She diffed it first. Then, instead of stopping at "it never ran," we asked what it could have reached, and that question found a leak that had been open to everyone since September at the latest.

The intruder got nothing we can find. The checking got us a fix we needed anyway. And the receipt we like least is the first one: the fix for the hole they used was nine months old, and production didn't have it. This ended as well as it did because we looked.