Podman on Shane's Personal Blog

How a Courier Knows Where to Run

Thu, 14 May 2026 00:00:00 +0000

If you’re new, the story so far — in What a Switch Actually Does I admitted I’d been faking my way through networking and finally understood what’s happening inside one virtual conference room. In The Art of Buying a Toaster I traced a full web request from a man clicking Buy on his couch to a small database far away and back.

This post is about something I cheated on in post 2.

The Cheat

In the toaster story I said “the Sea of Couriers” and waved my hand.

A vast, restless relay of runners between Jack’s home and the shop. Each one knows only the next runner on the route.

I made it sound poetic and you let me and we moved on. The story had momentum and a cat in it.

But after I finished writing that post, that one line started bothering me. Each one knows only the next runner.

Wait — how, exactly?

If you were the very first courier — standing on Jack’s doorstep when Browser-Bot pushed the satchel into your hand — how would you know which way to run? The address on the satchel said 93.184.216.34. You have never been to 93.184.216.34. You don’t have a map of the world in your pocket. You have two feet.

So which direction?

I had assumed, for years, that somewhere in the middle of the internet there was a clever piece of infrastructure that did know. The map lived at the top. The little couriers asked the big infrastructure where to go. I never poked at that assumption because nothing in my work had ever forced me to.

When I finally poked, the answer was so much weirder, simpler, and more humbling than I had braced for that I’m going to spend the whole post on it.

A Train Station in Tokyo

The cleanest version of the answer I can give isn’t technical. It’s about something every traveller has done.

You’re standing in a train station in Tokyo. You speak no Japanese. You have a paper map. You point at it and ask the nearest stranger — fumbled syllables, hopeful expression — “Eiffel Tower? Paris? France?”

The stranger looks at the map. Then at you. Then bursts out laughing, because of course she has no idea how to get from Tokyo to the Eiffel Tower. She lives here. But she’s kind and doesn’t want you to feel stupid, so she points at a sign and says, in slow careful Japanese you mostly parse: “Information desk. Third floor. They will help.”

You climb three flights of stairs. The man at the information desk doesn’t know how to get to the Eiffel Tower either. He has a binder. He flips it open, runs his finger down a column, looks up at you, and says: “For international travel — Narita Airport. Take the JR Narita Express, platform 4. Sixty minutes.”

You take the train.

At Narita, an Air France agent checks her terminal, smiles, hands you a boarding pass: “You want Paris–Charles-de-Gaulle. Flight 275, departs 21:35, gate 47.”

At Charles-de-Gaulle, a taxi driver outside the terminal nods when you say “Tour Eiffel,” and just drives. He’s done this drive ten thousand times.

When the taxi pulls up in front of the tower, you step out, tilt your head back, and there it is.

Now ask yourself this. At any point along that journey, did anyone know the way from the Tokyo train station to the Eiffel Tower?

No. Not the kind stranger. Not the information desk. Not even the airline agent. Each of them knew one hop closer than you did. Each of them delegated upward to someone with a slightly bigger frame of reference. The route from you to the tower existed — but no single mind ever held it. The route was distributed across the five strangers you spoke to, none of whom were in charge.

This is, without changing a single load-bearing element, exactly how the internet routes a packet from your doorstep to a server you’ve never heard of.

Your Home Router Is the Kind Stranger

The first courier on Jack’s doorstep — the one I waved at — is your home router. The little plastic box that’s been quietly serving your family’s WiFi for the last four years.

Your home router knows two things. Two. That’s it.

Every device on your home network. Your laptop, your phone, your smart fridge, your printer — anything plugged in or connected to the WiFi. If a parcel is addressed to one of those, the home router walks it there directly.
One phone number. Its supervisor’s. For every parcel not addressed to your home, it does exactly one thing: forwards the parcel up to the supervisor and washes its hands. Whatever that parcel is, wherever it’s going — it’s no longer the home router’s problem.

That second rule has a formal name. It’s called the default route, sometimes called, with mild gallows humour, the gateway of last resort. When I first heard those words I assumed they referred to some exotic configuration for niche cases. They don’t. The default route is the entire personality of your home router.

You can check this for yourself. If you SSH into your router (or peek at most home routers’ admin pages) and ask what its routing table looks like, you will see approximately three lines of text. Two of them you can ignore. The third is the default route. That is your home router. That is what the box streaming your Netflix actually does.

I cannot overstate how much this shrank the internet in my head when I first understood it. I had been picturing a clever piece of infrastructure. I was picturing the kind stranger in Tokyo.

The Supervisor Is Also Mostly Ignorant

The supervisor — your ISP’s edge router — is a slight upgrade. It has a real, if modest, map. The map covers its territory: the few thousand local addresses it serves, plus a handful of direct lines to other networks it’s cut deals with. (Big content networks like Netflix and Cloudflare have direct peering arrangements with major ISPs, which is why Netflix loads so fast — your ISP’s map has a special shortcut that says “for Netflix, take this side door.”)

If the address is on the map, the supervisor dispatches it. If it’s not, the supervisor does what the home router did. Forwards it up to her supervisor.

Same pattern. Slightly bigger map. Same single fallback.

This pattern repeats. Every level has a slightly more elaborate map. Every level has the same fallback. The whole hierarchy is held together by one tiny humble rule, repeated everywhere: I’ll handle my own area and pass the rest up.

And here is the part that, when it hit me, I had to stand up and walk around the apartment for a minute.

The exact same trick is running inside your laptop. Inside Frieda’s container, from post 2 — the little frontend in apartment 10 — her tiny routing table looks like this:

10.88.0.0/16 dev eth0 # the corridor — local
default via 10.88.0.1 # else → Mama Kernel

That’s it. Two lines. Same shape as the home router. Same shape as the ISP’s edge. One local rule, one default route. Frieda is doing — at the scale of one container — exactly what your home router does at the scale of your house, and what your ISP does at the scale of a neighbourhood.

The internet isn’t running on different mechanisms at different scales. It’s running on the same tiny mechanism, recursively, like a tree where every leaf has the same DNA as every branch. The home router is not a smaller version of a Big Important Internet Thing. The home router and the Big Important Internet Thing are the same shape.

Where the Buck Has to Stop

But you can’t have a tree where everyone delegates upward without someone, somewhere, at the very top, who does not delegate. There is no one above the top.

That top layer exists. It’s called the default-free zone — the DFZ. Backbone routers run by Tier-1 carriers and at the world’s internet exchange points. Their job is to know enough that they never have to say “I don’t know, ask above me.” Because there is no above.

I had imagined, when I first heard “Tier-1 carriers” and “default-free zone,” some kind of throne room. A great central server humming gently in a vault, holding the map. The official map. The one everyone else was downloading copies of.

There is no throne room. There is no central map.

What there is, instead, is a particular kind of conversation that happens, continuously, between every pair of neighbouring backbone networks.

The Conversation

The conversation goes like this.

Picture two operators, each sitting at her own desk. Both work for backbone networks — different companies, different countries, doesn’t matter. Each has a notebook open in front of her.

The first operator reads from her notebook: “From my side, I can reach the address range 8.8.8.0/24 — those are Google’s servers. And 1.1.1.0/24 — that’s Cloudflare. And these other three thousand ranges. Here’s the full list.”

The second operator writes everything down. Then she reads from her notebook: “Got it. From my side: I can reach 93.184.216.0/24 — those are some big websites. And these other two thousand ranges. Here’s mine.”

They both close their notebooks. They thank each other politely. They walk back to their own offices and update their own networks with what they learned.

That’s the conversation. That is all the conversation is.

They don’t consult a master. They don’t agree to anything except what each of them claims to be able to reach, and they take each other’s word for it. Then their other peers come asking the same question, and each operator passes along everything she just learned, plus the note “I heard this from her, by the way.”

This conversation, repeated across every backbone operator on Earth, builds a kind of distributed encyclopedia of how to reach every address range on the internet. The encyclopedia doesn’t live in any one place. Each operator has her own copy, slightly different from her neighbour’s, but mostly in agreement.

The protocol that runs this conversation is called BGP — the Border Gateway Protocol. The technical name for “a backbone operator” is an Autonomous System, abbreviated AS. Each AS has a number. Google is 15169. Cloudflare is 13335. Your ISP almost certainly has one. There are about 75,000 of them.

When I first learned that BGP — the thing holding the internet together — is structured gossip, I had a small moment of disbelief. That’s it? That’s what’s underneath? Yes. That’s it. The most important protocol on the internet is two strangers comparing notebooks, repeated everywhere.

What “Distributed” Actually Means

I want to underline something here because I had this part wrong for a long time.

I had carried, vaguely, the idea that there were two kinds of internet things. The cute distributed-feeling things — peer-to-peer file sharing, blockchain, the cottage projects I’d read about — ran on gossip. The serious, centralised, load-bearing things — the internet itself, the Tier-1s, the infrastructure my company depended on — ran on databases. I imagined a difference of seriousness. Gossip for the toys, real systems for the load-bearing stuff.

The internet runs on gossip. The serious thing is the cute thing. There is no master copy of the routing table. There is no central authority you can call to ask “what is the canonical path to 8.8.8.8?” The path is whatever the gossip currently says it is. If the gossip is briefly wrong — if some operator misspeaks, or her notebook gets garbled — the internet briefly routes wrong, and then the gossip corrects itself, and the world keeps turning.

The Tier-1 operators are big. They are economically powerful. They are not, however, in charge of routing. They are participants. If one of them disappeared overnight, the others would re-sync their notebooks from the remaining peers and traffic would re-route. It has happened. It will happen again. The internet was not designed to require any of its participants in particular.

There is one place where centralisation does live, and I should be honest about it: address allocation. Who is allowed to claim the range 8.8.8.0/24 in her notebook is not gossip-determined. There’s a real tree of authorities for that — at the top, an organisation called IANA. IANA hands huge chunks of address space to five regional registries (ARIN for North America, RIPE for Europe, APNIC for Asia, and so on). The regional registries hand smaller chunks to ISPs. The ISPs hand smaller chunks to companies. That part is a tree, complete with paperwork.

But allocation and routing are different problems. Allocation is “who’s allowed to say they own this range?” Routing is “how do I actually get there?” The first is centrally administered. The second is gossiped between equals. They sit on top of each other in a way I had collapsed in my head for years.

A Quiet Algorithmic Surprise

There’s one more thing I want to mention, because it surprised me.

I had a half-memory from undergrad of a clever pathfinding algorithm — something with a heuristic, A*-style. I was sure routing used something like it. The internet is, after all, the world’s largest graph. Surely some elegant shortest-path algorithm is running somewhere?

There is — but only inside a single AS. Not between them.

Inside a single autonomous system — say, all the routers within Google’s network — every router runs Dijkstra’s algorithm over a freshly-flooded map of the local topology. A* is the heuristic-enhanced version of Dijkstra you might be remembering; same family. Inside an AS, you really do find the shortest path. The protocol family that does this is called OSPF (or its cousin IS-IS), and right now, on tens of thousands of routers around the world, Dijkstra is running. The exact clean graph-search you imagined.

Between ASes, though — between Google’s network and Cloudflare’s — there is no shortest-path calculation. BGP doesn’t compute distance. BGP picks paths by business policy. Each operator’s notebook is annotated with her bosses’ preferences: “prefer routes through our cheap peer over our expensive transit provider.” Those preferences dominate path selection.

This means the route a packet actually takes between two points on the internet is not necessarily the geographically shortest one. It’s the cheapest-and-contractually-permitted one. A packet from Spain to Brazil might legitimately go through London because the Spanish operator has a sweetheart deal with a British peer and a punishing one with a French peer. The map of the internet is, in this sense, a financial document as much as it is a topological one.

I found this deeply uncomfortable when I realised it. I had imagined the internet as a pure physical structure — bones and veins. The actual structure is bones plus contracts. It is a network of physical wires held in place at the joints by negotiation. Not necessarily worse, but aesthetically different from what I’d been using.

The Aha

Here is what I now think the picture is.

The internet works because nobody tries to know the whole map. Every node — from the cheapest home router to the largest Tier-1 — plays the same little game: I know my immediate neighbours, and I delegate the rest. The whole vast distributed system is built out of one humble move, repeated billions of times, with structured gossip at the top to keep everyone in rough agreement.

The internet is not a cathedral with a divine architect. The internet is a forest of strangers asking each other for directions, and somehow it works. What I find quietly beautiful about this is that the design choice — delegate, don’t accumulate — was not chosen because it was elegant. It was chosen because the alternative was unbuildable. There was no way to make any single brain big enough to hold the whole map and keep it current as the world changed. So the engineers built no brain at all. They built only the protocol for two strangers to compare notebooks and trust each other’s word.

The next time you open a browser tab — your couch, your half-finished coffee, a casual click — there’s a fact you can hold onto. Somewhere in the middle of that request, a row of strangers each admitted they didn’t know where it was going and passed it to a neighbour. Then that neighbour did the same. Then another. The packet found its way home not because anyone planned the route, but because every node along the way was honest about the limits of what it knew.

I had spent years quietly assuming the internet was a triumph of centralised intelligence. It’s the opposite. It’s a triumph of cooperative ignorance.

That is what your internet bill pays for.

What I Still Don’t Understand

Two things, both unsettling.

First — what happens when one of the operators lies? What if an AS announces, by accident or on purpose, “hi, I can reach all of Google’s address ranges,” and the neighbours believe her, and the lie propagates outward? I have a dim memory of this happening to YouTube once. Something about Pakistan, an entire afternoon, a country accidentally swallowing a popular website’s traffic. I think I have the pieces now to actually understand how that worked. And, more uncomfortably, why it’s hard to prevent without giving up the gossip-based design that makes the rest of the system work.

Second — I keep promising namespaces. Frieda still has an apartment that I have not, technically, explained how exists. One kernel. Multiple parallel networks. Each container with its own private 127.0.0.1 that doesn’t trample anyone else’s. I waved at this in post 1 and again in post 2. I am, frankly, embarrassed about it.

I’ll get there. One of these is the next post. I’m not going to decide which until I sit down to write it.

See you then.

(Written by Human, improved using AI where applicable.)

The Art of Buying a Toaster

Wed, 13 May 2026 00:00:00 +0000

If you’re new, the story so far — in What a Switch Actually Does I admitted I’d been faking my way through networking for years, dug into what a switch actually does, and finally understood why a listener bound to 127.0.0.1 is invisible to a container connecting via 10.88.0.1.

For everyone else: this post is a slight detour, and on purpose.

The Embarrassment

Somewhere in the middle of writing post 1, I tried to explain what I was learning to a friend who doesn’t write code. We were having dinner. She’d politely asked what I’d been geeking out about all week.

I made it about three sentences in before her eyes did the thing.

You know the thing. The micro-glaze. The “I love you and I am still here, but I am no longer parsing English.” I’d already said “TCP” and “the kernel” and “bridge interface” and I was about to say “MAC address,” and I caught myself and stopped.

Here’s what hit me. I could now explain a switch. I could explain a virtual bridge. I could explain why my Portainer container couldn’t see my SSH tunnel. But if my friend had asked me — what actually happens, from start to finish, when she clicks Buy on a website? — I would have started flailing. I had pieces. I didn’t have a story.

So I sat down to write one. A real story. With characters. With a setting. With stakes. No jargon. The kind of story I could tell her over the rest of dinner.

The decoder ring — what every character “really” is in computer terms — is at the end. Read the story first. If you’ve never thought about how a website actually works, that’s perfect. Especially then.

The Cast

It helps to know the small cast before we begin. They’re going to come and go quickly.

Jack. A man at home, shopping online. He wants a toaster. This is his entire characterisation.

Browser-Bot. Jack’s little assistant. Lives in his laptop. Writes notes, packs them into parcels, hands them to couriers. Faithful, fussy.

The Sea of Couriers. A vast, restless relay of runners between Jack’s home and the shop. Each one knows only the next runner on the route. They never read the notes; they only pass them along.

The Front Gate. A small, stoic doorperson at the entrance of an office building somewhere far from Jack. Its entire job is to accept parcels addressed to this building and refuse all others. It cannot read.

Mama Kernel. The office manager. Knows every door, every desk, every resident. She sits in a back office with three things on her table: a directory of every department in the building, a rulebook for rewriting addresses, and an enormous ledger where she writes down every parcel that came in and what she did to it. Calm, unflappable, has seen everything.

Pre-Clerk and Post-Clerk. Two assistants who work in pairs. The Pre-Clerk meets every incoming parcel at the door and sometimes rewrites the address on the outside. The Post-Clerk does the same for outgoing parcels — and he keeps a copy of every rewrite his colleague made, so he can put things back exactly as they were on the way out.

The Hallway Concierge. Runs the inside corridor of the building. Knows every resident by sight. His only superpower is delivering a parcel from one apartment door to another — never up, never down, just sideways along his hallway.

Frieda the Frontend. A friendly resident on the corridor. Apartment 10. She runs a kind of front desk: she greets every visitor’s parcel, looks at what it’s asking for, and either answers it herself or politely forwards it down the hall to whoever can.

Brutus the Backend. Frieda’s neighbour. Apartment 11. Gruff bookkeeper. Will not talk to outsiders. Will only accept parcels handed to him by Frieda. Does the actual work — opening ledgers, debiting accounts, marking inventory.

The Parcels. Every message in this story is shaped like a Russian nesting doll. A letter inside an envelope inside a packet inside a delivery satchel. Each layer is addressed to a different recipient, for a different leg of the journey. We will watch them get unwrapped and rewrapped many times.

That is everyone. Now to the story.

Act One — The Click

Jack wants a toaster.

He has, in fact, wanted a toaster for several weeks. The old one finally died last Sunday, dramatically, in a small puff of smoke that made the cat leave the room. He has been putting this off because online shopping always feels like more of a commitment than it ought to. But today is the day.

He scrolls. He picks one. He clicks Buy.

In the corner of Jack’s laptop, Browser-Bot springs into action. He’s a tidy little creature, all bow-tie and clipboard. He scribbles a short note — barely a sentence: “this customer would like to buy one toaster” — and begins to wrap it for the road.

First, the note goes inside a small envelope, addressed in Browser-Bot’s neat handwriting. Then that envelope goes inside a larger packet, with a different address on it — the address of the shop, far away. Then the whole packet goes inside an outer satchel, with yet a third address on it — this one for the very first relay courier waiting outside Jack’s front door.

Three layers. Three addresses. Each one meant for a different leg of the journey. Browser-Bot has done this a thousand times today already. He hands the satchel out the door.

Act Two — Across the Sea of Couriers

A relay runner is already waiting. She takes the satchel, glances at the outer address, and runs.

She runs to the next runner, who tears off the outer wrapping — it had served its purpose — looks at the next address inside, wraps that inner packet in her own fresh satchel addressed to the next relay over, and hands it on.

This goes on for a long time.

Every runner along the way does the same thing: tear off the outer wrapping that was only meant for the last leg, look at the address on what’s inside, wrap it fresh for the next leg, send it along. The inner packet — the one with the actual shop’s address on it — never gets opened. It is too sacred. It is meant to travel.

If you could speed up time and watch from above, you’d see a glowing point hopping from runner to runner across a continent, leaving a trail of discarded satchels in its wake. The note inside has not moved relative to its packet. The packet has not moved relative to itself. Only the outermost wrapper keeps getting reborn.

Eventually, after what is in human terms a tiny fraction of a second, the packet reaches the office building it was always destined for. The last relay courier hands it across to the building’s Front Gate, gives a half-salute, and runs off into the night.

Act Three — The Onion Unwrapped

The Front Gate is uninterested in the contents of the satchel. The Front Gate is only interested in one thing: the name written on the outermost layer. It is the building’s name. The Gate accepts the parcel. The Gate’s job is done.

The parcel is conveyed into the back office, where Mama Kernel is waiting.

She picks it up gently. She tears off the outer satchel — that was only ever for the last leg of the road. She lays the inner packet on her table. The address on that is the building’s, all right, but written in a more permanent kind of way. Good. For us.

She is about to unwrap the next layer when the Pre-Clerk appears at her elbow.

“This one’s for Apartment 10,” he says. “Sales inquiry. Frieda’s department.”

Mama Kernel nods. The Pre-Clerk takes a pencil and, very gently, edits the address on the packet — changing the building’s name to Frieda’s exact apartment number. He turns to his ledger and notes carefully: Parcel #8472, originally addressed to the building, redirected to Apartment 10. Remember to reverse this on the way back. He underlines remember three times. He is very serious about his job.

Mama Kernel takes the relabelled packet and consults her directory. Apartment 10 — that’s down the corridor, the Concierge’s territory. She wraps the packet one more time, in a small in-building courier sleeve addressed by name to Frieda herself, and hands it through her window to the Hallway Concierge.

Act Four — Down the Corridor

The Concierge has, by now, memorised the face of every resident on his corridor. Frieda the Frontend, Apartment 10 — of course, of course. He whisks the parcel down the hall and slides it through the slot in her door.

Frieda is, as ever, at her front desk in a sunny mood. She unwraps the parcel — courier sleeve off, the building-layer wrapping off, the original travel packet off — and unfolds the small inner envelope and finally reads the note inside.

“This customer would like to buy one toaster.”

“Right!” Frieda says, brightly, to nobody. “That’s Brutus’s department.”

She produces a clean piece of paper and writes her own note: “a customer would like to buy one toaster — please process.” She wraps it in her own small envelope, addresses the envelope to Apartment 11 next door, wraps that in her own little courier sleeve, and slides the whole thing back out under her door for the Concierge.

This is important. Frieda did not forward the original parcel. She wrote a new parcel, with a new address, asking Brutus for the same thing on her behalf. The original parcel sits on her desk like a receipt. She’ll need it again in a minute.

The Concierge takes Frieda’s new parcel — three short steps down the hall — and slides it under Brutus’s door.

Brutus, in Apartment 11, does not look up from his ledger. He hears the parcel arrive. He sighs. He unwraps it. He reads. He grumbles. “Toaster. Customer. Fine.”

He turns to his great accounting book. He opens it. He debits one toaster from the inventory. He writes a note next to Jack’s name. He stamps the whole thing CONFIRMED in red ink. Then he writes a tiny reply — “order received, will ship, ID #42” — wraps it in a small envelope addressed to Frieda, and slides it back into the hall.

Concierge. Three steps. Frieda’s slot.

Frieda reads Brutus’s reply, smiles, and now turns back to the parcel she had set aside. She picks up her pen and writes a reply of her own, addressed back to the original sender — Jack — saying “order confirmed, your toaster is on its way, thank you for shopping with us.” She wraps it in the same nesting style she received: envelope inside packet inside courier sleeve. She slides it under her door.

Act Five — Home

The reply travels in reverse, exactly the way the request came. Concierge to Mama Kernel. Mama Kernel consults her directory: the sender lives way out beyond the building’s walls, so she’ll need to send it out via the Front Gate.

But before she can wrap it for outbound travel, the Post-Clerk appears at her elbow. He flips open his ledger to the page about parcel #8472.

“Don’t forget,” he says. “When this came in, my colleague rewrote the address from the building’s name to Apartment 10. We need to reverse that now. Frieda’s apartment number should not appear on this parcel as it leaves — to the outside world, the reply came from the building, not from any one apartment.”

He pencils in the correction. He closes the ledger.

Mama Kernel wraps the now-correctly-addressed packet in a fresh outer satchel and hands it to the Front Gate, who passes it to the first relay runner waiting outside. The Sea of Couriers reverses the relay across the continent, satchel by satchel, until at last a single runner sprints up Jack’s driveway and slips the parcel under his door.

Browser-Bot receives it, unwraps it ceremonially, smooths out the reply note, and reads.

“Order confirmed. Your toaster is on its way. Thank you for shopping with us.”

In Jack’s browser, the Buy button quietly turns into a green checkmark and changes its name to ✓ Ordered. Somewhere, a cat looks up from a sunny patch on the rug. Jack smiles.

He thinks: that was so easy.

The Decoder Ring

It was not, in fact, easy.

Here is what each character secretly was. Read it the way you’d read the credits at the end of a play and realise you’ve been watching something more elaborate than you thought.

Jack and his Browser-Bot. A user at a computer, and the web browser running on that computer. The browser is the program that translates human intent — “I clicked Buy” — into the actual sequence of messages a server needs to receive.

The note Browser-Bot wrote, and the nesting wrappers. This is the heart of the whole story, so I’ll spend a moment on it.

Every message that crosses the internet is built like a nesting doll, with each wrapper meant for a different audience. The innermost note is the actual content — in our case, “buy one toaster.” That’s called an HTTP request, the language web browsers and web servers use to talk to each other.

Around that, Browser-Bot wraps a TCP segment — a wrapper that adds reliability features (sequence numbers, retransmission, confirmation receipts) so that if any part of the journey loses the parcel, it can be detected and resent. TCP is what makes the internet reliable; without it, every web page would be a coin flip.

Around that, an IP packet — the layer that carries the actual end-to-end address. The IP packet says this is for the shop’s public address. The IP packet is the one wrapper that, like in the story, mostly never gets opened or rewritten as it crosses the continent. It travels end to end.

Around that, an Ethernet frame — the outermost satchel. The frame only knows about the next single hop. It’s addressed to the next runner, not to the final destination. At every hop along the way, the frame gets torn off and a new one written for the next hop.

That nesting — letter inside TCP inside IP inside Ethernet — is the famous “OSI layered model” most of us were taught in school as seven boxes to memorise. It is, in practice, just this: a parcel inside a parcel inside a parcel, each for a different audience along the route.

The Sea of Couriers. Routers across the public internet. Each router only knows the next hop; together they form a relay across continents. The fact that the inner IP packet never gets rewritten — only the outer Ethernet frame, fresh at every hop — is what makes the internet actually work. The IP packet survives end to end while the wrappers around it are disposable.

The Front Gate. The host’s network card — the NIC — facing the outside world. On a Linux box this interface is usually called eth0. Its job is exactly as described: accept frames addressed to us, drop the rest, hand the accepted ones up to the kernel.

Mama Kernel. The Linux kernel itself. She owns every interface in the building, every routing decision, every address rewrite, every connection in flight. The directory she consults is the kernel routing table; the rulebook beside it is iptables / netfilter; the great ledger of every connection she’s tracking is the conntrack table.

Pre-Clerk and Post-Clerk. These are the iptables hooks called PREROUTING and POSTROUTING. The rewriting they do is NAT — Network Address Translation. The specific kind of rewriting in our story (an incoming connection getting redirected to an internal apartment number) is called DNAT (Destination NAT). The reason Post-Clerk has to reverse the rewrite on the way out is so the outside world only ever sees the building’s public address, never the internal apartment numbers.

The Hallway Concierge. A virtual bridge inside the kernel. On a Linux box running containers, it’s commonly named docker0 or podman0. It is, behaviourally, exactly the kind of switch we explored in post 1 — a thing that delivers messages between residents of one corridor by name, by lookup or by flooding, and does nothing else. The fact that it’s software and not a metal box in a rack is a detail; the algorithm is identical.

Frieda the Frontend and Brutus the Backend. Two containers, each living in its own isolated apartment. Frieda might run, say, an nginx server that handles incoming web traffic. Brutus might run a database or an order-processing API. They’re plugged into the same internal corridor (the bridge) but isolated from each other — and from the rest of the building — in a way we still have to talk about.

The moment Frieda decided to “write a new parcel to Brutus” instead of “forwarding the original” is exactly the moment a real reverse proxy makes a fresh request to an upstream service. It is two separate conversations stitched together, not one conversation passed through. That distinction matters more than it sounds; we’ll come back to it.

Apartment 10 and 11, the corridor, the building. These are the IPs you’ve been seeing all over post 1 and this one. Frieda’s apartment number is 10.88.0.10. Brutus’s is 10.88.0.11. The corridor is the 10.88.0.0/16 subnet. The host — Mama Kernel’s own seat at the corridor’s table — is 10.88.0.1. The building’s public address is whatever IP the outside world sees the host as; that one varies.

The conversation between Frieda and Brutus that the outside world never saw. This is the inter-container traffic on the bridge. Notice it crossed no NAT, no routing decision worth speaking of, no outside interface. Two apartments, one corridor, Concierge passing parcels. Cheap, fast, invisible from outside. This is why container-to-container traffic on a bridge is so quick — the kernel basically never has to leave the lower layers.

The two ledgers being kept in lockstep. That’s connection tracking — conntrack — the kernel feature that remembers every active connection and what rewrites were applied to it, so reply traffic can be reversed perfectly even though the rewriting was applied asymmetrically. Without this, no NAT scheme on Earth would work. With it, the entire modern internet (where every home router does this, all day, for every device on your home network) functions invisibly.

The Aha

When my friend at dinner heard “the internet,” she pictured, I think, a kind of pneumatic tube. You put a thing in one end, it comes out the other. Magic in the middle.

The middle is not magic. The middle is a relay of small, dumb, identical actors, each doing one tiny job per parcel, none of them aware of the others, none of them aware of the contents. The “intelligence” lives at the two ends — Browser-Bot wrapping the parcel with the right addresses on each layer, and the receiving server unwrapping them and replying. The middle is just a sequence of helpful strangers passing notes.

Inside the office building, the relay continues. Mama Kernel is more sophisticated than any one courier — she can rewrite addresses, route between corridors, keep ledgers — but she is still, fundamentally, a series of small mechanisms in sequence. Receive at the door. Strip the outer wrapper. Look at the address. Maybe rewrite it. Look up where it goes. Wrap it again. Send it out the right inner door. That’s a pipeline, not a wizard. You can hold the whole thing in your head once you stop expecting it to be one big magic step.

This was the second click for me, after the switch one. The switch unlocked what happens at one moment in time, in one place. The toaster story unlocked what happens across an entire journey. They go together.

What I Still Don’t Understand

There’s a moment in the story I quietly waved past.

When the parcel arrived at Frieda’s apartment, I said her “local kernel” peeled the wrappers, her own door slot received the courier sleeve, her own desk held the original packet. Frieda has, apparently, her own little internal world: her own front door, her own loopback to herself, her own apartment number, her own desk, her own everything.

But Frieda isn’t a separate computer. Frieda is just a process running on the same host as Mama Kernel. There is only one kernel in this building. There is only one set of hardware.

So how does Frieda’s apartment exist? How does it have its own internal addresses, its own loopback, its own private experience of the corridor that doesn’t trample on Brutus’s right next door? How does the same kernel run multiple parallel network worlds side by side, each believing it’s alone?

The answer is the feature I waved at in post 1 without explaining. It’s called a network namespace, and once it clicks, the entire reason containers feel like “their own little computers” snaps into focus. It’s also the foundation under VPNs, under sandboxing, under a lot of zero-trust networking — and under the bug I’m slowly explaining across this whole series.

I’ll get there. I promise.

See you next post — whatever rabbit hole I fall into first.

(Written by Human, improved using AI where applicable.)

What a Switch Actually Does

Tue, 12 May 2026 00:00:00 +0000

I was trying to do something I’d done a dozen times.

Set up Portainer — the Docker management UI — on my laptop. Have it manage a remote server through a Cloudflare tunnel so nothing is exposed publicly. The remote machine runs a Portainer agent; the laptop runs the Portainer server; they talk over SSH wrapped in Cloudflare Access.

I knew the moving parts. I’d touched all of them before. SSH tunnels, Docker, Portainer, Cloudflare — none of this was new.

It didn’t work.

The agent on the remote was alive. The UI on my laptop was alive. Both endpoints were reachable when I poked them from a terminal. But when I clicked Connect in the Portainer UI, I got the cleanest, most damning error in all of networking:

Get "https://host.containers.internal:19001/ping": dial tcp 10.88.0.1:19001: connect: connection refused

“Connection refused.” The kind of error that means something is rejecting your call. Not “no answer,” not “no route,” but a deliberate door slam.

I stared at it for an embarrassing amount of time. Every link in the chain checked out individually. The agent was listening. The SSH tunnel had bound its port. curl from the laptop to the tunnel worked. The Portainer container itself was healthy.

Each piece was fine. The error said something about an interface I couldn’t even fully explain — 10.88.0.1 — and a hostname I’d typed for years without really thinking about what it meant — host.containers.internal.

That was the moment.

The Honest Admission

I’d been operating on memorized commands.

I knew ssh -L forwards a port. I knew docker run -p 9001:9001 “exposes” a port. I knew 127.0.0.1 was “localhost.” I’d typed all of this hundreds of times and it had always just worked. And when it didn’t, I’d usually fix it by stack-overflowing my way to a different flag combination that did work, without ever understanding why.

I learned the OSI model in college as seven boxes to memorize for the exam. Physical, data link, network, transport, session, presentation, application. I could’ve named them on a quiz. I could not have told you what the difference between a bridge and a router is. I’d worked professionally for years without filling that gap, because the abstractions sit on top of each other so cleanly that you can build whole careers on the top layer and pretend the bottom six aren’t there.

But they are there. And as our company has scaled, the asks have started reaching deeper into the cake — reverse proxies, streaming, multi-region failover, zero-trust networking, “why does this work in dev but not in staging.” Every one of them pokes at exactly the gaps I’d been papering over.

A connection refused error from inside a container, against a port my own laptop was clearly serving, was just the latest version of the same gap. The bug wasn’t in the tools. The bug was in me.

So I decided to actually learn it.

This is post one of an open-ended series. The plan isn’t to write a textbook — the world has plenty of those, and I wouldn’t be the right author. The plan is to dig into the layer of the stack that’s directly behind whatever I’m currently confused by, until the confusion lifts, then move to the next layer. The Portainer bug is the spine. Every post returns to it.

By the end of the series, the error message above will read like a sentence in plain English, not an incantation.

This post is about the first thing I had to understand: what is actually happening at the lowest layer that’s relevant to my bug. What is 10.88.0.1? Why is it different from 127.0.0.1? What is a “bridge,” really?

“Wait, but how does the network even work inside my laptop?”

Here’s the part that hit me first.

The agent on the remote was running on port 19001. My SSH tunnel forwarded my laptop’s port 19001 to the remote’s port 19001. I could curl https://127.0.0.1:19001/ping from my laptop terminal and get a clean reply.

The Portainer Server, running inside a container on the same laptop, could not reach the same address. It got “connection refused.”

Same machine. Same kernel. Same port number. Different result.

How is that possible?

The naive mental model — the one I’d been carrying — is that a computer has a network. There’s “my machine,” and either something is reachable from it or it isn’t. If curl works, anything else running on the same box should be able to do the same thing.

That model is wrong, and it has been wrong since well before Docker existed. A modern Linux box is not a single network. It is a network of networks, stitched together inside one kernel. Containers, VMs, VPNs, even ordinary SSH tunnels — they all rely on this fact. You just don’t have to think about it until something breaks weirdly.

To make any sense of why my container couldn’t see what my terminal could, I had to start with the dumbest, most fundamental piece of the puzzle: the virtual switch sitting between my container and the rest of the machine.

I’m going to tell you about it as a story. Bear with me. There’s a small cast.

A Small Cast

Margaret. A receptionist with infinite patience and a single notebook. She stands in the middle of a meeting room. Her entire job, all day, is to deliver paper notes between the people sitting at the table.

Alice, Bob, Carlos. Three coworkers seated at the table. Each one wears a name badge clipped to their shirt. The badges are unique, machine-printed, and frankly weird-looking — they say things like aa:42:81:9c:0e:33. Nobody’s ever asked why. They came that way.

The Notebook. Margaret’s notebook, in which she writes down — and only ever writes down — who sits at which seat. Nobody helps her fill it in. She fills it in herself, just by watching.

The Notes. Folded paper. Each note has a name badge on the outside (who it’s for) and some content on the inside (what’s being said). Margaret only ever looks at the outside.

That’s the cast for the room. Now the story.

The Conference Room

Alice has just started at the company. Today is her first day. She walks in, picks an empty seat — seat 3 — and sits down.

Margaret notices nothing in particular. She has no idea who Alice is yet. The notebook says nothing about Alice.

A few minutes later, Alice writes a quick “good morning” note to Carlos and hands it to Margaret. Margaret takes the note, reads the destination — Carlos’s name badge — and thinks: “Carlos? Carlos. Seat 7.” She delivers it to seat 7.

But before she walks off, Margaret also does something almost imperceptible. She glances at the sender’s name on the outside of the envelope — Alice’s badge — and the seat she just picked the note up from. She writes a line in her notebook:

Alice (badge: aa:42:81:9c:0e:33) — seat 3.

Now she knows.

Ten minutes later, Bob (seat 5) wants to send Alice a note. He writes it, addresses it to Alice’s badge, and hands it to Margaret. Margaret flips open her notebook, looks up Alice, sees “seat 3,” and walks the note over. Done.

This is the calm, well-run version of Margaret’s day. Everyone she’s seen, she knows. Looking up a seat is a one-second flip of a page.

But what about the moment before she knew Alice existed? What if Bob had wanted to send Alice a note five minutes into Alice’s first day, while she was still settling in, before Alice had sent so much as a single note Margaret could learn from?

Margaret would flip open the notebook. Alice’s name wouldn’t be there. She’d shrug, walk around the room, and hand a copy of the note to everyone except Bob. The way she sees it: I don’t know who Alice is, so the simplest thing is to give everyone a copy and let them sort it out. Whichever one is Alice will read her copy and react; everyone else will glance at the name on the outside, see it’s not theirs, and drop the copy in the bin.

That’s it. That’s Margaret’s entire job. Two behaviours.

If she knows the seat: look up, deliver. Fast, quiet, no fuss.
If she doesn’t know the seat: hand a copy to everyone. Crude but effective.

And every note she sees — whether it’s the one she’s looking up or the one she’s about to flood — she also passively learns from. “Note came up from seat 3, sender badge aa:42:81:9c:0e:33” — into the notebook it goes. Plug a new person in, have them send one note, and Margaret knows where they are. Forever. She’s never wrong, because she only writes down what she’s actually seen with her own eyes.

That’s the whole conference room. Two mechanisms, one self-filling notebook, zero knowledge of anything above her own little job.

What Margaret Was, In Computer Terms

Time to break the spell, briefly, and say who Margaret really is.

Margaret is a switch. In software form she’s also called a bridge, but the algorithm is identical. Hardware switches sit in rack-mounted metal boxes in data centres. Software bridges sit as data structures in your laptop’s kernel. They behave the same.
The seats are ports on the switch — the actual jacks where Ethernet cables plug in. Not the TCP ports you might be thinking of. Different concept, same word. Sorry.
The name badges are MAC addresses. Every network card on Earth has one, burned in at the factory. They look exactly as weird as Margaret’s: six pairs of hex characters separated by colons.
Margaret’s notebook is the switch’s MAC address table. The mapping of “this badge lives at this seat.”
Behaviour 1 (lookup-and-deliver) is called forwarding.
Behaviour 2 (hand-to-everyone) is called flooding.
The trick where Margaret writes down what she observes is called MAC learning, and it is genuinely one of the most elegant designs in the whole stack. The switch self-configures from passively watching traffic. No DHCP, no config file, no setup wizard. Plug a device in. Have it transmit one frame. The switch knows where it is.

That last point matters. Margaret doesn’t read the notes. Margaret only looks at the outside. She has no idea what protocol the senders are using, what language they’re writing in, what they’re talking about. She doesn’t need to. Forward or flood. Repeat.

This is the reason physical switches can forward millions of frames per second in dedicated silicon. There’s almost nothing to compute. Read source badge, update notebook. Read destination badge, look up or flood. Done.

Bob Asks for Someone He Doesn’t Know

Now the obvious next question. The one that snagged me.

Okay — Margaret can lookup-or-flood. Fine. But how does Bob know what badge to write on the outside of his envelope in the first place? When I type an IP address into a browser, my computer eventually has to know which physical network card owns that IP. Who answers that question?

Back to the room.

Today, Bob has a problem. He’s been asked to send a note to “the person who handles toaster orders.” He has no idea which of his coworkers that is. He doesn’t know their name badge. He doesn’t know their seat.

Margaret can’t help directly. Her notebook maps badges to seats; it doesn’t tell you what people do.

But Bob knows a trick.

He writes his note. The note inside says, “Hi — I’m Bob, badge bb:11:22:33:44:55, seat 5. Whoever handles toaster orders, please write back and tell me your badge.” Then on the outside of the envelope, in the destination field, instead of writing a specific person’s badge, he writes:

FF:FF:FF:FF:FF:FF

This is a special, reserved name badge. Nobody actually wears it. It’s a magic phrase. The whole room knows, by convention, that this badge means “everyone.”

He hands the envelope to Margaret.

Margaret looks at the destination: FF:FF:FF:FF:FF:FF. She flips open the notebook. It’s not there — of course it isn’t, nobody wears that badge. So she does the only thing she ever does when a destination isn’t in her notebook: she walks around the room and hands a copy to every seat except Bob’s.

Notice what just happened. Margaret didn’t do anything special. She didn’t “recognise” Bob’s question or “decide” to broadcast. She got a destination she couldn’t find in the notebook and fell into her flood behaviour. The smart move was on the sender’s side. Bob picked the magic badge because he wanted Margaret to flood for him.

Everyone gets a copy. Most people glance at the inside, see “toaster orders” doesn’t apply to them, and drop their copy in the bin. But Alice — who, it turns out, is in charge of toaster orders — reads her copy carefully. Then she writes back to Bob, this time with the destination set to Bob’s actual badge, content: “Hi Bob, I handle toasters, my badge is aa:42:81:9c:0e:33.”

Margaret, in delivering Alice’s reply back to seat 5, also passively writes Alice into her notebook (if she hadn’t already). Bob now knows Alice’s badge. Margaret now knows Alice’s seat. Everyone has learned what they need to know from a single exchange.

In computer terms: that exchange is called ARP — Address Resolution Protocol. The magic badge FF:FF:FF:FF:FF:FF is called the broadcast MAC address. The whole protocol is just: send a question to everyone using the magic badge, the right device replies, you remember its real badge, and now you can talk directly.

This was the bit that clicked hardest for me. ARP isn’t a third behaviour on top of forward-and-flood. It’s a clever use of the flood behaviour, triggered by the sender choosing the magic destination. The switch is along for the ride. It can’t tell ARP apart from regular traffic, and it doesn’t need to.

Trigger	Why it floods	Who decided
Unknown destination badge	Switch has no entry, falls back	The switch (forced by ignorance)
Destination badge is `FF:FF:FF:FF:FF:FF` (broadcast)	Sender wrote “everyone” on envelope	The sender (intentional)

(One small footnote that delighted me when I learned it: FF:FF:FF:FF:FF:FF being “all 1s” isn’t arbitrary. The lowest bit of the first byte of a MAC address is reserved to mean “this is a group address.” Hardware can check “is this a broadcast?” with a single-bit test on one byte — cheaper than a notebook lookup. Hardware-friendliness was baked into the protocol from day one.)

The Room Goes Virtual

Everything I just described is the world of a physical switch — a metal box in a rack, with Ethernet jacks on the front. Margaret is sitting in an actual room, with actual humans, passing actual paper.

The switch on my laptop is virtual. Linux runs it. The kernel calls these things bridges. Docker’s is named docker0; Podman’s is named podman0. Behaviourally, it is identical to the physical box: same notebook, same two mechanisms, same broadcast handling, same total ignorance of anything above its own little job. The difference is that Margaret has moved into the kernel, and the room around her is now made of data structures in memory rather than transistors and copper.

When a container starts, the kernel creates a virtual Ethernet cable — a veth pair. Think of it as a literal cable with two ends. One end gets moved into the container, where it shows up as eth0. The other end gets plugged into a seat in Margaret’s room. The kernel guarantees: whatever goes in one end comes out the other. That’s the entire abstraction.

And here is the part that, for me, was the unlock: the host is also seated in Margaret’s room. When Margaret’s room is first set up, the kernel gives the host its own seat at the table — its own badge, its own IP. Margaret does not know or care which seat belongs to the host and which belong to containers. They’re all just people with name badges, and she passes notes between them by the only rules she has.

Docker and Podman, at this layer, are doing three boring things:

Ask the kernel to create the room (the bridge).
Plug containers in and out via veth pairs as they start and stop.
Make sure the host gets its own seat at the table.

The kernel does the actual switching. Docker and Podman are just running the room.

And here is where my bug starts to make sense: 10.88.0.1 is the host’s seat at Margaret’s table in Podman’s room. It’s the IP of the host’s port on podman0. From inside a container, that IP is “the host, as seen from inside this room.” That is what host.containers.internal is a friendly DNS name for. It’s not a magic concept — it’s the host’s seat at this particular meeting room. (Docker is the same idea with different numbers: 172.17.0.1 is where the host sits in docker0’s room.)

But the Building Has Many Rooms

If the host is sitting in Margaret’s room with an IP of 10.88.0.1, you might be wondering: what about 127.0.0.1? What about good old localhost?

Localhost is a different room.

Imagine the office building has, somewhere down the hall, a tiny private room where each company runs its own internal mail circle. The room only has one seat in it — yours — and a separate receptionist, Lorraine. Lorraine’s whole job is even simpler than Margaret’s: when you hand her a note, she walks two steps and hands it back to you. That’s it. There’s nobody else in the room. Notes you write to yourself in Lorraine’s room never leave.

Lorraine is the loopback interface — lo, address 127.0.0.1. Every Linux box has one. It’s the room where a program talks to itself.

The host has a seat in both rooms. The host sits with Margaret (10.88.0.1) and also sits with Lorraine (127.0.0.1). Different rooms, different receptionists, different addresses.

Here’s the part I’d never properly understood. When a program on the host opens a network listener — call it a doorman waiting to accept connections — it has to decide which room’s door to put the doorman in front of. Bind your doorman to 127.0.0.1 and the doorman is standing in Lorraine’s room; anyone wanting to talk to him has to be sitting in Lorraine’s room too. Bind your doorman to 10.88.0.1 and he’s standing in Margaret’s room. Bind him to 0.0.0.0 (“any room”) and he stands in every room the host has a seat in.

This is exactly where my bug lived.

When my SSH tunnel set up its listener — “please accept incoming connections on port 19001 and forward them out to the remote server” — it bound the doorman to 127.0.0.1. The doorman set up shop in Lorraine’s room.

When I, on the host, ran curl https://127.0.0.1:19001/ping from a terminal, I was sitting in Lorraine’s room asking for the door to be answered. The doorman was right there. He answered. curl worked.

But when the Portainer container — sitting at the table in Margaret’s room — sent its request to the host’s interface on the bridge (10.88.0.1:19001), the request arrived through Margaret’s room. The container knocked on a door in Margaret’s room. The doorman wasn’t there. He was over in Lorraine’s room. The host had no listener on this interface for this port. Connection refused.

The fix, once I could see it, was idiotic in its smallness. Bind the SSH tunnel’s listener to 10.88.0.1 instead. Or 0.0.0.0. Anything that put the doorman in the room where the container’s knock would actually land.

But getting to “idiotic in its smallness” required me to know there were multiple rooms at all. Which required me to know what a bridge was. Which required me to know that Margaret only does two things. Which is why this is post one.

So, Did It All Make Sense?

It made more sense. Not all sense.

I now understood why 10.88.0.1 mattered and why it isn’t the same as 127.0.0.1. I could see Margaret’s room in my head — a virtual conference room with my container at one seat and the host at another, Margaret passing notes by name badge, doing forward-or-flood and nothing else. I could see why a listener bound to one interface — a doorman in one room — was invisible to traffic arriving from another. I could see why ARP wasn’t magic — just Bob using the magic “everyone” badge to ask a question, and the switch faithfully flooding because that’s all it knows how to do.

But a much weirder question had taken its place. The whole story above assumes that a container has its own network stack to plug into Margaret’s room. The container is, supposedly, just a process running on my laptop’s kernel. So how does a process get its own network card? Its own IP? Its own seat in Lorraine’s private little room that isn’t the host’s seat in Lorraine’s room?

And come to think of it — when I’d been confidently saying “containers are isolated,” what did I actually mean by isolated? Isolated from what? Using what mechanism? Where in the kernel does that isolation live?

I had filled in one gap and immediately found a bigger one underneath it.

Network namespaces are coming — the kernel feature that lets a single Linux machine pretend to be many independent networks, all running side by side, all believing they have their own 127.0.0.1. It’s the foundation of containers, VPNs, and a surprising amount of modern infrastructure — and once it clicks, the veth pair I waved at above suddenly stops being a metaphor and starts being a piece of obvious plumbing.

See you next post.

(Written by Human, improved using AI where applicable.)