In the late 1990s, I was at Microsoft and was handed a problem with no obvious answer: build the infrastructure for Windows Update.
Not improve it. Not iterate on it. Build it, from nothing.
Windows Update went on to become one of the ten most visited sites on the internet. What we built wasn’t just a way to deliver downloads. It was an early version of what we’d now call cloud-scale service delivery: global distribution, active-active redundancy, and always-on availability, at a volume the industry didn’t have a playbook for yet.
I think about that work a lot, because almost everything I’ve done since has been shaped by what I learned there.
If you’re curious how Microsoft approaches Windows Update delivery today, its Windows deployment documentation is a good place to start.
When there’s no playbook, you write it
We were operating at a scale where the conventional wisdom didn’t exist yet. There were no best practices for always-on, globally redundant web infrastructure at that volume. We had to figure it out, document it, and operate it, all at the same time.
I worked with F5 on server clustering and DNS enhancements that became foundational to how we kept the service available worldwide. We developed datacenter optimization practices that were adopted broadly across Microsoft’s web infrastructure. And we designed an active-active model that could absorb the traffic spike of a major software release or update without falling over.
What that embedded in me was a specific discipline: when there’s no playbook, you build one. You reason from first principles, you test your assumptions under real load, and you write down what you learn so the next person doesn’t start from zero.
Scale changes the failure calculus
At small scale, a failure is an inconvenience. At Windows Update scale, a failure was a news story. If it didn’t work, the operating system appeared to be broken.
When millions of people depend on your infrastructure on the day a major update ships, the blast radius of any incident isn’t a team problem. It’s a company problem, a customer problem, and a press problem. That reality forces a different kind of rigor.
You don’t get to say “we’ll fix it in production.” You don’t get to learn from a live outage the way you might in a smaller shop. You design for contained failure from the start, because the cost of skipping it is measured in headlines, not help desk tickets.
That mindset, designing for contained failure rather than optimistic success, is something I’ve brought to every program since, from hospital cutovers to M&A integrations.
The hidden lesson: documentation is infrastructure
This part doesn’t get enough credit in conversations about large-scale programs: the operational documentation we built was as important as the architecture itself.
An active-active datacenter model is only as reliable as the runbooks that govern it. A globally distributed service is only as resilient as the team’s shared understanding of what to do when something behaves strangely at 2 a.m., in a time zone that’s nobody’s time zone.
Documentation isn’t overhead. It’s the connective tissue that lets people who didn’t build a complex system operate it safely.
I’ve carried that into every environment since: healthcare IT, logistics platforms, SaaS, and global carve-outs. The programs that run well are the ones where the knowledge lives in the system, not just in the heads of the people who built it.
What I’d tell my younger self
Build things that outlast you. Write down what you learn. Design for the failure modes, not just the happy path.
And when someone hands you a blank slate and says, “We need this to work at scale, and there’s no playbook,” don’t treat that as a problem.
That’s the most interesting work there is.
The programs that run well are the ones where the knowledge lives in the system, not just in the people who built it.
Andrew Weinberg is a Principal Technical Program Manager with 25+ years in healthcare IT, enterprise infrastructure, SaaS, and M&A integration.
Get new field notes by email
Practical lessons on cutovers, M&A integration, and keeping high-stakes programs on track. New posts most Tuesdays.
No spam. Unsubscribe anytime.
