Keeping code and systems safe

The situation

46 of my projects run in containers on one server, and I'm the only person looking after them. I don't read every line of what the AI writes, so the checks have to run without me.

What I did

The guards are code, run by about 35 timers. When a check fails it blocks the work or raises a task with the bot that owns the project. Anything that really needs me pages my phone.

• Image scanning

  Every running image is scanned every 24 hours. A known-exploited flaw in a public service pages my phone. A critical or high flaw with a fix becomes a task for the bot that owns the project.

• Base images

  Four shared base images are rebuilt from upstream every Sunday. All 46 projects that run in containers are built on them.

• Host checks

  A daily check of the operating system advisories against the installed packages and the kernel that is booted. I added it after a serious kernel CVE went unnoticed for two weeks.

• Supply chain

  pnpm everywhere, a 48-hour wait on new package releases, unusual sub-dependencies blocked, and a pre-push gate that refuses drift from the pinned versions.

• Test gate

  A push is blocked unless the tree is clean and the test status is green and signed against the code being pushed. If the test count drops, someone has to find out why.

• Backups

  Nightly encrypted database backups, kept for 30 days, with a copy offsite. Restores are tested.

• Certificates

  One canonical certificate replaced about 24 hand-copied ones. It renews weekly, and a check looks at what each of 67 endpoints is serving.

• Release gate

  The only way private code becomes public. Six steps, then I type the approval.

October 2026

A certificate authority was compromised. That evening a scan of every container showed which ones trusted the affected roots, and about 70 containers were rebuilt that night. The final scan found none that still accepted the forged chain. There is now a trust-store audit on a timer that raises tasks by itself.

The result

Most of this I never see. Backups run, images get rebuilt, findings go to the bot that can fix them, and my phone tells me when something needs me. You can't mitigate completely, but you can put things in place to improve the situation, and human-written software has issues too.

What a team can take from it

I think the checks have to run on their own and block when they fail. I learnt that from the AIs. They would delete a test that was stopping them committing their code, so I had to put things in place to stop that, and the same thinking went into everything on this page.