All posts

A Patch Is Done When the Fleet Proves It

Use build records and production inventory to verify a security patch reached every service, including scheduled jobs, background workers, and regions missing telemetry.

A coral route splits toward three stations. One branch stops short of its station, representing a workload whose update is unverified.

Follow the patch into production.

Imagine a team gives itself until noon to roll out a Ruby security fix. The web deployment goes green at 11:40. An import worker still runs yesterday's build, and one region has stopped reporting container status. The release channel says the update shipped. The on-call engineer still has to update the worker and find out what's running in that region.

That distinction matters under Google Project Zero's new disclosure policy. On November 13, Natalie Silvanovich announced that the team would normally publish vulnerability details 90 days after reporting the issue to the vendor. The policy removes the old post-fix waiting period and schedules shorter disclosure windows. Silvanovich calls the time between release of a fix and installation by the user the "patch gap." Project Zero's disclosure policy

To verify a container rollout, compare your workload inventory and inspected builds with what's running in production. That comparison shows which services need an upgrade and where evidence is missing.

Start with what runs in production

Ruby maintainer Yusuke Endoh announced fixes for seven vulnerabilities in the interpreter. The fixes shipped in Ruby 3.3.14, 3.4.13, and 4.0.10. On November 20, Cloudflare's Daniele Molteni, Pawel Klimek, and Denzil Correa advised teams fixing Ruby2Shell to rebuild container images and restart background workers. They also called out self-hosted products that ship with their own copy of Ruby. Ruby security announcement · Cloudflare's Ruby2Shell analysis

Start with a list of production services and scheduled jobs, then look up the software in the images they use. Include self-hosted products your team has to upgrade, even when their internals live outside your repositories. For each entry, record who owns the upgrade, what runs today, and where the next instance will pull its build. That last detail matters for a nightly import or a standby region that may have nothing running at the moment you check.

Track each workload by region or environment. A billing service running in two regions needs two confirmations. In each region, count the service as verified once every active replica runs an approved build and the deployment template points to one. Reconcile the list against your deployment inventory or platform accounts so a missing region stays visible when its telemetry drops out.

Tie the fix to a specific build

Inspect the image you plan to ship. Check which Ruby binary or package it contains. Save that result with the digest. Check the fixed package for your distribution if it supplies Ruby. If a product bundles Ruby, consult its vendor's advisory for the release that contains the fix. In each case, keep enough detail for the next on-call engineer to tell which evidence applies to which build.

Kubernetes' image documentation explains how a mutable tag can leave Pods running different versions when the registry changes what the tag points to. Deploying by digest pins the image identity. The digest gives you a stable ID. The inspection record shows what it contained when you approved it. Kubernetes image documentation

On Kubernetes, read ContainerStatus.imageID for the running container. Resolve the runtime's reference to the digest in your inspection record. If you ship for more than one architecture, use the digest for the platform-specific image that actually ran. For future starts, check that the deployment template or scheduled job points to an approved digest. Kubernetes notes that editing a CronJob updates future Jobs while ones already started continue under the old configuration. For a dormant scheduled job, record the template change and check the digest on its next run or on a controlled run in the same environment, if the job can run safely outside its usual schedule. Kubernetes Pod API reference · Kubernetes multi-architecture images · Kubernetes CronJob documentation

Keep missing evidence in the count

Suppose 24 of the 26 expected workload and environment pairs are confirmed. One worker is still on the old build. One region is missing current observations. Report 24 verified, one pending upgrade, and one unknown. Assign someone to investigate the region with missing data and set a time to check again. If a service is temporarily isolated or covered by a compensating control, record the protection, who owns it, when it expires, and what evidence will close the patch ticket.

Cloudflare's Ruby2Shell rules illustrate the distinction. Its web application firewall can reduce exposure for traffic it inspects. The authors recommend upgrading Ruby because traffic outside that coverage can still reach affected software. Record the filter as temporary protection and track when the fixed runtime is in use. Cloudflare's Ruby2Shell analysis

NIST's patch management guide recommends verifying that the update took effect and continuing to monitor it. Keep the verified digests after the rollout. Use them to detect a rollback or a restore from an older backup. NIST SP 800-40 Rev. 4, sections 2.3.3 and 2.3.4

Record how long it takes to get a deployable build after a fix is available for your distribution. Then measure how long it takes to confirm the new build is running in production. If you're waiting on a vendor release before you can start, track that wait as well.

Try this on one runtime before the next urgent advisory. Pick a service with a web deployment and a scheduled worker. Ask someone who didn't ship the current release to find both digests and show which Ruby versions the builds contain. Have that person account for every production environment. If they need to ask the person who deployed it, record what they couldn't find. Fill in the missing records before the next patch.