Three Times The Measurement Killed The Plan

I asked a simple question about a project I maintain, which publishes a security newsletter and the static site behind it: can we make the website deploys faster? They took about ten minutes, they ran several times a day across two brands, and ten minutes is long enough that you stop watching and go do something else, which is its own kind of cost.

The number I started from came from a GitHub Actions step log. One step called "Deploy committed publication" ran nine and a half minutes, another called "Build Cyber artifact" ran ten and a half, and from those two numbers I concluded the obvious thing, which is that the deploy is essentially the site build and the site build is essentially page count. The site prerenders a few thousand pages and most of them are archive, meaning published newsletter issues and tool pages that will never change again, so the plan wrote itself: stop rebuilding the archive on every deploy, copy the previous render forward, and gate the reuse behind content hashes so a template change still repaints everything. It was a good plan. It was approved. It was also wrong three separate times, and each time the thing that proved it wrong took a few minutes to run.

Every number I had about the deploy came from a log that could only see the outside of the thing I wanted to change.

The wrong mental model

The plan rested on a chain of assumptions that each sounded reasonable and none of which I had checked, and they are worth laying out because I suspect most optimization plans have a chain like this in them somewhere.

Every link in that chain is an inference from an aggregate. A CI step log gives you a number for a step, and a step is whatever somebody happened to put in one block of YAML, so it tells you nothing about the shape of the work inside it. I had been reasoning about the interior of a box from the label on the outside.

Measurement one: the build was a quarter of the deploy, not all of it

Before optimizing anything I added phase timers, which is a context manager wrapping each named step of the build and deploy, printing the elapsed seconds and banking them into the JSON receipt the deploy already writes for CI to read. It is about thirty lines of Python and it is the least interesting code in the project.

@contextlib.contextmanager
def phase(name: str):
    start = time.monotonic()
    try:
        yield
    finally:
        elapsed = time.monotonic() - start
        _PHASE_TIMINGS.append((name, elapsed))
        print(f"  ⏱  {name}: {elapsed:.1f}s", flush=True)

It records in a finally on purpose, because a phase that died after eight minutes is the single most useful number in a failed deploy and swallowing it on the way out of an exception throws that away. Then I ran one real deploy and printed the phases slowest first, which took about twelve minutes of wall clock including the wait, and here is what came back.

Deploy phases (total 703.3s):
    180.9s   25.7%  build:astro
    147.5s   21.0%  publish:api-data
    130.2s   18.5%  publish:subscriber-ereader-bundles
     81.1s   11.5%  push:media
     66.2s    9.4%  publish:report-artifacts
     40.5s    5.8%  render:pdfs
     18.4s    2.6%  build:finalize
     14.4s    2.0%  pages:push

The build was 181 seconds of a 703 second deploy, so about a quarter. The three publishing phases together were 344 seconds, which is roughly half, and nothing in the approved plan touched any of them. The plan's entire ceiling was a fraction of a number that was itself only a quarter of the problem, and I had been about to spend days on it.

A measurement whose only job is to tell you where the time goes is worth building before the fix, because it is the cheapest thing that can change your mind.

Measurement two: the archive was worth 88 seconds, not ten minutes

With the build reduced to a quarter of the deploy, the next question was how much of that quarter the archive reuse could actually recover, and the honest way to answer it is to build the site with those pages simply not built. So I patched three getStaticPaths functions to return two entries each behind an environment variable, ran a build, and threw the patch away, which took about six minutes for both builds.

Dropping 2,163 pages saved 88 seconds, which means perfect reuse of every archive page, with a manifest and content hashing and a fallback path and a byte-identity gate, has a ceiling of 88 seconds on a 703 second deploy. Twelve percent, for the most complicated and highest-risk change on the list, where the failure mode is serving a stale page as current and nothing downstream can detect it.

Meanwhile the two publishing phases I had not looked at yet turned out to be worth more than that between them and carried no staleness risk at all, so I did those first. One was uploading about 120 small JSON objects to S3 one at a time, which is round-trip latency in series and became a bounded thread pool, and the other had a genuine bug I will come back to.

When a plan's ceiling is measurable, measure it before you build the plan, because the ceiling is often smaller than the risk.

Measurement three: the expensive pages were not the ones I thought

I still had 88 seconds on the table and a decision to make about whether to keep going, so I ran the same trick again with more resolution, emptying one route at a time instead of all three together. Three builds, about nine minutes.

That table is the whole article. Eleven hundred report pages cost one second between them and nine hundred tool pages cost seventy-six seconds, and when two sets of pages of nearly identical size differ by seventy-five seconds you no longer have a page-count problem, you have a bug. A page-count problem is linear and boring; this is something else.

It took about four minutes of reading to find it. Every tool page called three functions in its own body, and not one of them cached anything.

const issues = publicIssues() as HistoryIssue[];
const watchlist = loadWatchlist();
const tool = toolDetail(issues, slug!, watchlist);

loadWatchlist() reads and parses a 700-tool YAML file from disk. publicIssues() re-derives all 99 archived issues through a sort and a mapping function. toolDetail() groups every entry of every issue into a Map of about 900 tools, then walks that Map linearly, slugifying each name, to find the single tool the page is about. All of it, 935 times, to answer 935 questions that each concern one tool. Quadratic in the size of the roster, and the roster grows every week, so it had been getting slowly worse for a year while looking like a static site that simply had a lot of pages.

The fix is memoization and it is unglamorous.

+let watchlistCache: WatchlistTool[] | null = null;
+
 export function loadWatchlist(): WatchlistTool[] {
+  if (watchlistCache) return watchlistCache;
   try {
     const text = readFileSync(new URL(rel, import.meta.url), 'utf8');
-    return parseWatchlist(text);
+    watchlistCache = parseWatchlist(text);
+    return watchlistCache;

Three caches of that shape, plus a WeakMap keyed on the issue array's identity for the grouping so that a caller passing a different list gets grouped afresh rather than served somebody else's answer, and a slug-keyed lookup replacing the linear scan. What makes it safe is that all three are build inputs, meaning static module imports and a file that nothing writes while a build is running, so a repeated call could only ever have produced the same answer it produced the first time.

A cost that scales with something other than the work you asked for is a bug, and the way you find it is by measuring two comparable things and noticing they aren't.

Nine hundred pages costing seventy-six seconds while eleven hundred cost one second is not a performance characteristic, it's a defect wearing one.

Side by side

The plan and what actually shipped, which share a goal and nothing else:

And the build itself:

I also tried the thing you are supposed to try, which is Astro's build.concurrency, because the build ran at 124% CPU on a machine with 28 cores and that looks exactly like a parallelism problem. Four workers gave 173.7 seconds and eight gave 169.3, against a 179 second baseline, so it bought three to five percent and I reverted it. It was worth six minutes to find out, and it is worth saying out loud that the obvious knob was nearly useless while the unglamorous cache was nine times.

The fix, in order of what it cost to find

1. Phase timers in the deploy, about thirty lines, one deploy to get a baseline. This overturned the premise that the build was the deploy, and everything after it depended on knowing the real split.

2. Parallel S3 uploads for the API data publish, collecting the objects instead of uploading them inline and flushing them through a sixteen-worker pool. 147 seconds to roughly ten, with no ordering dependency to break because nothing in that prefix reads one key through another.

3. An existence check on the report artifacts, which is the bug I owe you. That step renders report bundles with a headless browser and caps itself at 25 attempts per deploy, and the comment above the cap says an artifact already uploaded is skipped so the next deploy moves on to ones that have none. Nothing checked. The cap counted attempts, the index order is stable, and the key is a content hash, so every deploy re-rendered the same first 25 reports forever and the remaining 1,106 rows were never reached and never would be. It now asks the store which keys are missing and spends the cap on those, and building a report to compute its key had been re-reading the whole issue archive per row, which memoizing per window took from 187 seconds to nine.

4. Memoizing the three build-input reads, which took the site build from 178.9 seconds to 19.7.

The deploy should now land around 540 seconds instead of 703, and the plan I was approved to build contributed nothing to that.

Verification, because a nine times speedup should make you suspicious

A change that fast is more likely to be a change that skipped work than a change that stopped repeating it, so the gate was byte-identity. Build the site with the memoization, save dist, stash the change, build again, and diff the trees.

The diff came back with exactly one difference across 3,279 pages, on one line of one file, and it was this:

A: <span class="row-released" title="2026-08-28T19:43:03Z"> yesterday </span>
B: <span class="row-released" title="2026-08-28T19:43:03Z"> 2d ago </span>

That is a relative date computed from the clock, and the two builds straddled the rounding boundary between one day and two. Nothing to do with the change, and worth knowing for its own sake, because it means this build is not byte-reproducible anyway and a content-hash reuse cache would have been fighting that as well.

What generalizes

Build the instrument before the fix. The phase timers changed the plan on their first run, and they cost less than an hour. Anything you are about to spend days on deserves an hour spent making sure it is the right days.

An aggregate is not a measurement. A CI step timing tells you about a block of YAML somebody wrote, not about the work inside it, and the same goes for a total request time, a job duration, or any number produced by a layer that cannot see the layer you want to change. If the number comes from outside the thing, it is a hint about where to instrument, not a result.

Measure the ceiling of a plan, not just the cost of the problem. Emptying a getStaticPaths to see what those pages are worth took six minutes and told me the most complex change on the list was capped at twelve percent. Most plans have an equivalent, meaning a crude way to simulate the best case by deleting the work rather than optimizing it, and it is nearly always cheaper than the real thing.

Compare two things that should cost the same and see if they do. One route with 1,129 pages cost a second and another with 935 cost seventy-six, and that comparison is what turned a performance question into a bug report. Absolute numbers tell you where the time is; ratios between comparable things tell you whether the time makes sense.

Prefer the fix with no new state. The plan needed a manifest, a version field, a fingerprint, a fallback path, and an escape hatch, and each of those is a thing that can be wrong later in a way nobody notices, since the failure mode of a stale cached page is a correct-looking page. The memoization added no state that outlives a single build, so there is nothing to invalidate and nothing to get stale.

Treat a per-item cap as a promise to make progress, and check that it does. A cap that counts attempts rather than completions, over an input in a stable order, with no check for work already done, is not spreading the work across runs; it is a treadmill, and it looks fine in the logs every single time.

The plan was wrong and the plan was still worth having

The honest part is that the approved plan was not stupid, and the person who approved it, which was me, had every reason to. Content-hash-verified page reuse is a real technique that real sites need, and if the build had genuinely been three minutes of unavoidable per-page rendering it would have been the right call. What it lacked was any evidence, and the evidence turned out to be about twenty minutes of builds away.

I should also say what this does not cover. The deploy is still around nine minutes of work I have not touched, including a hundred and thirty seconds of headless-browser rendering that is genuine backlog work and will drain on its own, and eighty seconds of media syncing against a store where nothing ever changes, which I suspect is another measurement waiting to embarrass me. The tool pages are twenty seconds now instead of seventy-six, but nothing about the architecture stops the next quadratic from moving in, and the thing that would, which is a build-time assertion that per-page work stays constant as the roster grows, does not exist.

The part I would keep if I could only keep one thing is the phase timers, which took an hour and are still the only reason I know what any of this costs.

The plan survived approval, review, and a written specification, and it did not survive the first twelve minutes of actually looking.