<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Osi-Model on Shane's Personal Blog</title><link>https://www.shanechang.com/tags/osi-model/</link><description>Recent content in Osi-Model on Shane's Personal Blog</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><lastBuildDate>Wed, 13 May 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://www.shanechang.com/tags/osi-model/index.xml" rel="self" type="application/rss+xml"/><item><title>The Art of Buying a Toaster</title><link>https://www.shanechang.com/p/networking-journey-buying-a-toaster/</link><pubDate>Wed, 13 May 2026 00:00:00 +0000</pubDate><guid>https://www.shanechang.com/p/networking-journey-buying-a-toaster/</guid><description>&lt;img src="https://www.shanechang.com/p/networking-journey-buying-a-toaster/cover.webp" alt="Featured image of post The Art of Buying a Toaster" /&gt;&lt;p&gt;If you&amp;rsquo;re new, the story so far — in &lt;a class="link" href="../networking-journey-portainer-and-switches/" &gt;What a Switch Actually Does&lt;/a&gt;
I admitted I&amp;rsquo;d been faking my way through networking for years, dug into what a switch actually does, and finally understood why a listener bound to &lt;code&gt;127.0.0.1&lt;/code&gt; is invisible to a container connecting via &lt;code&gt;10.88.0.1&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For everyone else: this post is a slight detour, and on purpose.&lt;/p&gt;
&lt;h2 id="the-embarrassment"&gt;The Embarrassment
&lt;/h2&gt;
&lt;p&gt;Somewhere in the middle of writing post 1, I tried to explain what I was learning to a friend who doesn&amp;rsquo;t write code. We were having dinner. She&amp;rsquo;d politely asked what I&amp;rsquo;d been geeking out about all week.&lt;/p&gt;
&lt;p&gt;I made it about three sentences in before her eyes did the thing.&lt;/p&gt;
&lt;p&gt;You know the thing. The micro-glaze. The &amp;ldquo;I love you and I am still here, but I am no longer parsing English.&amp;rdquo; I&amp;rsquo;d already said &amp;ldquo;TCP&amp;rdquo; and &amp;ldquo;the kernel&amp;rdquo; and &amp;ldquo;bridge interface&amp;rdquo; and I was about to say &amp;ldquo;MAC address,&amp;rdquo; and I caught myself and stopped.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s what hit me. I could now explain a switch. I could explain a virtual bridge. I could explain why my Portainer container couldn&amp;rsquo;t see my SSH tunnel. But if my friend had asked me — &lt;em&gt;what actually happens, from start to finish, when she clicks Buy on a website?&lt;/em&gt; — I would have started flailing. I had pieces. I didn&amp;rsquo;t have a story.&lt;/p&gt;
&lt;p&gt;So I sat down to write one. A real story. With characters. With a setting. With stakes. No jargon. The kind of story I could tell her over the rest of dinner.&lt;/p&gt;
&lt;p&gt;The decoder ring — what every character &amp;ldquo;really&amp;rdquo; is in computer terms — is at the end. Read the story first. If you&amp;rsquo;ve never thought about how a website actually works, that&amp;rsquo;s perfect. Especially then.&lt;/p&gt;
&lt;h2 id="the-cast"&gt;The Cast
&lt;/h2&gt;
&lt;p&gt;It helps to know the small cast before we begin. They&amp;rsquo;re going to come and go quickly.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Jack.&lt;/strong&gt; A man at home, shopping online. He wants a toaster. This is his entire characterisation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Browser-Bot.&lt;/strong&gt; Jack&amp;rsquo;s little assistant. Lives in his laptop. Writes notes, packs them into parcels, hands them to couriers. Faithful, fussy.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Sea of Couriers.&lt;/strong&gt; A vast, restless relay of runners between Jack&amp;rsquo;s home and the shop. Each one knows only the next runner on the route. They never read the notes; they only pass them along.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Front Gate.&lt;/strong&gt; A small, stoic doorperson at the entrance of an office building somewhere far from Jack. Its entire job is to accept parcels addressed to this building and refuse all others. It cannot read.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Mama Kernel.&lt;/strong&gt; The office manager. Knows every door, every desk, every resident. She sits in a back office with three things on her table: a directory of every department in the building, a rulebook for rewriting addresses, and an enormous ledger where she writes down every parcel that came in and what she did to it. Calm, unflappable, has seen everything.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Pre-Clerk and Post-Clerk.&lt;/strong&gt; Two assistants who work in pairs. The Pre-Clerk meets every incoming parcel at the door and sometimes rewrites the address on the outside. The Post-Clerk does the same for outgoing parcels — and he keeps a copy of every rewrite his colleague made, so he can put things back exactly as they were on the way out.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Hallway Concierge.&lt;/strong&gt; Runs the inside corridor of the building. Knows every resident by sight. His only superpower is delivering a parcel from one apartment door to another — never up, never down, just sideways along his hallway.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Frieda the Frontend.&lt;/strong&gt; A friendly resident on the corridor. Apartment 10. She runs a kind of front desk: she greets every visitor&amp;rsquo;s parcel, looks at what it&amp;rsquo;s asking for, and either answers it herself or politely forwards it down the hall to whoever can.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Brutus the Backend.&lt;/strong&gt; Frieda&amp;rsquo;s neighbour. Apartment 11. Gruff bookkeeper. Will not talk to outsiders. Will only accept parcels handed to him by Frieda. Does the actual work — opening ledgers, debiting accounts, marking inventory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Parcels.&lt;/strong&gt; Every message in this story is shaped like a Russian nesting doll. A letter inside an envelope inside a packet inside a delivery satchel. Each layer is addressed to a different recipient, for a different leg of the journey. We will watch them get unwrapped and rewrapped many times.&lt;/p&gt;
&lt;p&gt;That is everyone. Now to the story.&lt;/p&gt;
&lt;h2 id="act-one--the-click"&gt;Act One — The Click
&lt;/h2&gt;
&lt;p&gt;Jack wants a toaster.&lt;/p&gt;
&lt;p&gt;He has, in fact, wanted a toaster for several weeks. The old one finally died last Sunday, dramatically, in a small puff of smoke that made the cat leave the room. He has been putting this off because online shopping always feels like more of a commitment than it ought to. But today is the day.&lt;/p&gt;
&lt;p&gt;He scrolls. He picks one. He clicks &lt;strong&gt;Buy&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;In the corner of Jack&amp;rsquo;s laptop, Browser-Bot springs into action. He&amp;rsquo;s a tidy little creature, all bow-tie and clipboard. He scribbles a short note — barely a sentence: &lt;em&gt;&amp;ldquo;this customer would like to buy one toaster&amp;rdquo;&lt;/em&gt; — and begins to wrap it for the road.&lt;/p&gt;
&lt;p&gt;First, the note goes inside a small envelope, addressed in Browser-Bot&amp;rsquo;s neat handwriting. Then that envelope goes inside a larger packet, with a different address on it — the address of the shop, far away. Then the whole packet goes inside an outer satchel, with yet a third address on it — this one for the very first relay courier waiting outside Jack&amp;rsquo;s front door.&lt;/p&gt;
&lt;p&gt;Three layers. Three addresses. Each one meant for a different leg of the journey. Browser-Bot has done this a thousand times today already. He hands the satchel out the door.&lt;/p&gt;
&lt;h2 id="act-two--across-the-sea-of-couriers"&gt;Act Two — Across the Sea of Couriers
&lt;/h2&gt;
&lt;p&gt;A relay runner is already waiting. She takes the satchel, glances at the outer address, and runs.&lt;/p&gt;
&lt;p&gt;She runs to the next runner, who tears off the outer wrapping — it had served its purpose — looks at the next address inside, wraps that inner packet in &lt;em&gt;her own&lt;/em&gt; fresh satchel addressed to the next relay over, and hands it on.&lt;/p&gt;
&lt;p&gt;This goes on for a long time.&lt;/p&gt;
&lt;p&gt;Every runner along the way does the same thing: tear off the outer wrapping that was only meant for the last leg, look at the address on what&amp;rsquo;s inside, wrap it fresh for the next leg, send it along. The inner packet — the one with the actual shop&amp;rsquo;s address on it — never gets opened. It is too sacred. It is &lt;em&gt;meant&lt;/em&gt; to travel.&lt;/p&gt;
&lt;p&gt;If you could speed up time and watch from above, you&amp;rsquo;d see a glowing point hopping from runner to runner across a continent, leaving a trail of discarded satchels in its wake. The note inside has not moved relative to its packet. The packet has not moved relative to itself. Only the outermost wrapper keeps getting reborn.&lt;/p&gt;
&lt;p&gt;Eventually, after what is in human terms a tiny fraction of a second, the packet reaches the office building it was always destined for. The last relay courier hands it across to the building&amp;rsquo;s Front Gate, gives a half-salute, and runs off into the night.&lt;/p&gt;
&lt;h2 id="act-three--the-onion-unwrapped"&gt;Act Three — The Onion Unwrapped
&lt;/h2&gt;
&lt;p&gt;The Front Gate is uninterested in the contents of the satchel. The Front Gate is only interested in one thing: the name written on the outermost layer. It is the building&amp;rsquo;s name. The Gate accepts the parcel. The Gate&amp;rsquo;s job is done.&lt;/p&gt;
&lt;p&gt;The parcel is conveyed into the back office, where Mama Kernel is waiting.&lt;/p&gt;
&lt;p&gt;She picks it up gently. She tears off the outer satchel — that was only ever for the last leg of the road. She lays the inner packet on her table. The address on &lt;em&gt;that&lt;/em&gt; is the building&amp;rsquo;s, all right, but written in a more permanent kind of way. Good. For us.&lt;/p&gt;
&lt;p&gt;She is about to unwrap the next layer when the Pre-Clerk appears at her elbow.&lt;/p&gt;
&lt;p&gt;&amp;ldquo;This one&amp;rsquo;s for Apartment 10,&amp;rdquo; he says. &amp;ldquo;Sales inquiry. Frieda&amp;rsquo;s department.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;Mama Kernel nods. The Pre-Clerk takes a pencil and, very gently, edits the address on the packet — changing &lt;em&gt;the building&amp;rsquo;s&lt;/em&gt; name to &lt;em&gt;Frieda&amp;rsquo;s exact apartment number&lt;/em&gt;. He turns to his ledger and notes carefully: &lt;em&gt;Parcel #8472, originally addressed to the building, redirected to Apartment 10. Remember to reverse this on the way back.&lt;/em&gt; He underlines &lt;em&gt;remember&lt;/em&gt; three times. He is very serious about his job.&lt;/p&gt;
&lt;p&gt;Mama Kernel takes the relabelled packet and consults her directory. Apartment 10 — that&amp;rsquo;s down the corridor, the Concierge&amp;rsquo;s territory. She wraps the packet one more time, in a small in-building courier sleeve addressed by name to Frieda herself, and hands it through her window to the Hallway Concierge.&lt;/p&gt;
&lt;h2 id="act-four--down-the-corridor"&gt;Act Four — Down the Corridor
&lt;/h2&gt;
&lt;p&gt;The Concierge has, by now, memorised the face of every resident on his corridor. Frieda the Frontend, Apartment 10 — of course, of course. He whisks the parcel down the hall and slides it through the slot in her door.&lt;/p&gt;
&lt;p&gt;Frieda is, as ever, at her front desk in a sunny mood. She unwraps the parcel — courier sleeve off, the building-layer wrapping off, the original travel packet off — and unfolds the small inner envelope and finally reads the note inside.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&amp;ldquo;This customer would like to buy one toaster.&amp;rdquo;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;&amp;ldquo;Right!&amp;rdquo; Frieda says, brightly, to nobody. &amp;ldquo;That&amp;rsquo;s Brutus&amp;rsquo;s department.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;She produces a clean piece of paper and writes &lt;em&gt;her own&lt;/em&gt; note: &lt;em&gt;&amp;ldquo;a customer would like to buy one toaster — please process.&amp;rdquo;&lt;/em&gt; She wraps it in her own small envelope, addresses the envelope to Apartment 11 next door, wraps that in her own little courier sleeve, and slides the whole thing back out under her door for the Concierge.&lt;/p&gt;
&lt;p&gt;This is important. Frieda did not forward the original parcel. She wrote a &lt;em&gt;new&lt;/em&gt; parcel, with a new address, asking Brutus for the same thing on her behalf. The original parcel sits on her desk like a receipt. She&amp;rsquo;ll need it again in a minute.&lt;/p&gt;
&lt;p&gt;The Concierge takes Frieda&amp;rsquo;s new parcel — three short steps down the hall — and slides it under Brutus&amp;rsquo;s door.&lt;/p&gt;
&lt;p&gt;Brutus, in Apartment 11, does not look up from his ledger. He hears the parcel arrive. He sighs. He unwraps it. He reads. He grumbles. &lt;em&gt;&amp;ldquo;Toaster. Customer. Fine.&amp;rdquo;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;He turns to his great accounting book. He opens it. He debits one toaster from the inventory. He writes a note next to Jack&amp;rsquo;s name. He stamps the whole thing &lt;em&gt;CONFIRMED&lt;/em&gt; in red ink. Then he writes a tiny reply — &lt;em&gt;&amp;ldquo;order received, will ship, ID #42&amp;rdquo;&lt;/em&gt; — wraps it in a small envelope addressed to Frieda, and slides it back into the hall.&lt;/p&gt;
&lt;p&gt;Concierge. Three steps. Frieda&amp;rsquo;s slot.&lt;/p&gt;
&lt;p&gt;Frieda reads Brutus&amp;rsquo;s reply, smiles, and now turns back to the parcel she had set aside. She picks up her pen and writes a reply of her own, addressed back to the original sender — &lt;em&gt;Jack&lt;/em&gt; — saying &lt;em&gt;&amp;ldquo;order confirmed, your toaster is on its way, thank you for shopping with us.&amp;rdquo;&lt;/em&gt; She wraps it in the same nesting style she received: envelope inside packet inside courier sleeve. She slides it under her door.&lt;/p&gt;
&lt;h2 id="act-five--home"&gt;Act Five — Home
&lt;/h2&gt;
&lt;p&gt;The reply travels in reverse, exactly the way the request came. Concierge to Mama Kernel. Mama Kernel consults her directory: &lt;em&gt;the sender lives way out beyond the building&amp;rsquo;s walls&lt;/em&gt;, so she&amp;rsquo;ll need to send it out via the Front Gate.&lt;/p&gt;
&lt;p&gt;But before she can wrap it for outbound travel, the Post-Clerk appears at her elbow. He flips open his ledger to the page about parcel #8472.&lt;/p&gt;
&lt;p&gt;&amp;ldquo;Don&amp;rsquo;t forget,&amp;rdquo; he says. &amp;ldquo;When this came in, my colleague rewrote the address from the building&amp;rsquo;s name to Apartment 10. We need to reverse that now. Frieda&amp;rsquo;s apartment number should not appear on this parcel as it leaves — to the outside world, the reply came &lt;em&gt;from the building&lt;/em&gt;, not from any one apartment.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;He pencils in the correction. He closes the ledger.&lt;/p&gt;
&lt;p&gt;Mama Kernel wraps the now-correctly-addressed packet in a fresh outer satchel and hands it to the Front Gate, who passes it to the first relay runner waiting outside. The Sea of Couriers reverses the relay across the continent, satchel by satchel, until at last a single runner sprints up Jack&amp;rsquo;s driveway and slips the parcel under his door.&lt;/p&gt;
&lt;p&gt;Browser-Bot receives it, unwraps it ceremonially, smooths out the reply note, and reads.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&amp;ldquo;Order confirmed. Your toaster is on its way. Thank you for shopping with us.&amp;rdquo;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In Jack&amp;rsquo;s browser, the Buy button quietly turns into a green checkmark and changes its name to &lt;strong&gt;✓ Ordered&lt;/strong&gt;. Somewhere, a cat looks up from a sunny patch on the rug. Jack smiles.&lt;/p&gt;
&lt;p&gt;He thinks: &lt;em&gt;that was so easy&lt;/em&gt;.&lt;/p&gt;
&lt;h2 id="the-decoder-ring"&gt;The Decoder Ring
&lt;/h2&gt;
&lt;p&gt;It was not, in fact, easy.&lt;/p&gt;
&lt;p&gt;&lt;img src="https://www.shanechang.com/p/networking-journey-buying-a-toaster/how_a_web_request_travels.png"
width="1491"
height="1055"
srcset="https://www.shanechang.com/p/networking-journey-buying-a-toaster/how_a_web_request_travels_hu_9326d6dfef30a866.png 480w, https://www.shanechang.com/p/networking-journey-buying-a-toaster/how_a_web_request_travels_hu_4caa1cee948b196e.png 1024w"
loading="lazy"
alt="The full journey of one web request from Jack’s click to the response on his screen: packet nesting at the browser, routers rebuilding the Ethernet frame at every hop, the Linux host pipeline, the conntrack snapshot that lets replies undo the NAT, the frontend-to-backend hop that stays entirely on the podman0 bridge, and the symmetrical reply path back to the client."
class="gallery-image"
data-flex-grow="141"
data-flex-basis="339px"
&gt;
&lt;/p&gt;
&lt;p&gt;Here is what each character secretly was. Read it the way you&amp;rsquo;d read the credits at the end of a play and realise you&amp;rsquo;ve been watching something more elaborate than you thought.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Jack and his Browser-Bot.&lt;/strong&gt; A user at a computer, and the web browser running on that computer. The browser is the program that translates human intent — &amp;ldquo;I clicked Buy&amp;rdquo; — into the actual sequence of messages a server needs to receive.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The note Browser-Bot wrote, and the nesting wrappers.&lt;/strong&gt; This is the heart of the whole story, so I&amp;rsquo;ll spend a moment on it.&lt;/p&gt;
&lt;p&gt;Every message that crosses the internet is built like a nesting doll, with each wrapper meant for a different audience. The innermost note is the actual content — in our case, &amp;ldquo;buy one toaster.&amp;rdquo; That&amp;rsquo;s called an &lt;strong&gt;HTTP request&lt;/strong&gt;, the language web browsers and web servers use to talk to each other.&lt;/p&gt;
&lt;p&gt;Around that, Browser-Bot wraps a &lt;strong&gt;TCP segment&lt;/strong&gt; — a wrapper that adds reliability features (sequence numbers, retransmission, confirmation receipts) so that if any part of the journey loses the parcel, it can be detected and resent. TCP is what makes the internet &lt;em&gt;reliable&lt;/em&gt;; without it, every web page would be a coin flip.&lt;/p&gt;
&lt;p&gt;Around &lt;em&gt;that&lt;/em&gt;, an &lt;strong&gt;IP packet&lt;/strong&gt; — the layer that carries the actual end-to-end address. The IP packet says &lt;em&gt;this is for the shop&amp;rsquo;s public address&lt;/em&gt;. The IP packet is the one wrapper that, like in the story, mostly never gets opened or rewritten as it crosses the continent. It travels end to end.&lt;/p&gt;
&lt;p&gt;Around &lt;em&gt;that&lt;/em&gt;, an &lt;strong&gt;Ethernet frame&lt;/strong&gt; — the outermost satchel. The frame only knows about the &lt;em&gt;next single hop&lt;/em&gt;. It&amp;rsquo;s addressed to the next runner, not to the final destination. At every hop along the way, the frame gets torn off and a new one written for the next hop.&lt;/p&gt;
&lt;p&gt;That nesting — letter inside TCP inside IP inside Ethernet — is the famous &amp;ldquo;OSI layered model&amp;rdquo; most of us were taught in school as seven boxes to memorise. It is, in practice, just this: a parcel inside a parcel inside a parcel, each for a different audience along the route.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Sea of Couriers.&lt;/strong&gt; Routers across the public internet. Each router only knows the next hop; together they form a relay across continents. The fact that the inner IP packet never gets rewritten — only the outer Ethernet frame, fresh at every hop — is what makes the internet actually work. The IP packet survives end to end while the wrappers around it are disposable.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Front Gate.&lt;/strong&gt; The host&amp;rsquo;s network card — the &lt;strong&gt;NIC&lt;/strong&gt; — facing the outside world. On a Linux box this interface is usually called &lt;code&gt;eth0&lt;/code&gt;. Its job is exactly as described: accept frames addressed to us, drop the rest, hand the accepted ones up to the kernel.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Mama Kernel.&lt;/strong&gt; The Linux kernel itself. She owns every interface in the building, every routing decision, every address rewrite, every connection in flight. The directory she consults is the &lt;strong&gt;kernel routing table&lt;/strong&gt;; the rulebook beside it is &lt;strong&gt;iptables / netfilter&lt;/strong&gt;; the great ledger of every connection she&amp;rsquo;s tracking is the &lt;strong&gt;conntrack table&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Pre-Clerk and Post-Clerk.&lt;/strong&gt; These are the iptables hooks called &lt;strong&gt;PREROUTING&lt;/strong&gt; and &lt;strong&gt;POSTROUTING&lt;/strong&gt;. The rewriting they do is &lt;strong&gt;NAT&lt;/strong&gt; — Network Address Translation. The specific kind of rewriting in our story (an incoming connection getting redirected to an internal apartment number) is called &lt;strong&gt;DNAT&lt;/strong&gt; (Destination NAT). The reason Post-Clerk has to reverse the rewrite on the way out is so the outside world only ever sees the building&amp;rsquo;s public address, never the internal apartment numbers.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Hallway Concierge.&lt;/strong&gt; A &lt;strong&gt;virtual bridge&lt;/strong&gt; inside the kernel. On a Linux box running containers, it&amp;rsquo;s commonly named &lt;code&gt;docker0&lt;/code&gt; or &lt;code&gt;podman0&lt;/code&gt;. It is, behaviourally, &lt;em&gt;exactly&lt;/em&gt; the kind of switch we explored in &lt;a class="link" href="../networking-journey-portainer-and-switches/" &gt;post 1&lt;/a&gt;
— a thing that delivers messages between residents of one corridor by name, by lookup or by flooding, and does nothing else. The fact that it&amp;rsquo;s software and not a metal box in a rack is a detail; the algorithm is identical.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Frieda the Frontend and Brutus the Backend.&lt;/strong&gt; Two &lt;strong&gt;containers&lt;/strong&gt;, each living in its own isolated apartment. Frieda might run, say, an nginx server that handles incoming web traffic. Brutus might run a database or an order-processing API. They&amp;rsquo;re plugged into the same internal corridor (the bridge) but isolated from each other — and from the rest of the building — in a way we still have to talk about.&lt;/p&gt;
&lt;p&gt;The moment Frieda decided to &amp;ldquo;write a new parcel to Brutus&amp;rdquo; instead of &amp;ldquo;forwarding the original&amp;rdquo; is exactly the moment a real reverse proxy makes a fresh request to an upstream service. It is two &lt;em&gt;separate&lt;/em&gt; conversations stitched together, not one conversation passed through. That distinction matters more than it sounds; we&amp;rsquo;ll come back to it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Apartment 10 and 11, the corridor, the building.&lt;/strong&gt; These are the IPs you&amp;rsquo;ve been seeing all over post 1 and this one. Frieda&amp;rsquo;s apartment number is &lt;code&gt;10.88.0.10&lt;/code&gt;. Brutus&amp;rsquo;s is &lt;code&gt;10.88.0.11&lt;/code&gt;. The corridor is the &lt;code&gt;10.88.0.0/16&lt;/code&gt; subnet. The host — Mama Kernel&amp;rsquo;s own seat at the corridor&amp;rsquo;s table — is &lt;code&gt;10.88.0.1&lt;/code&gt;. The building&amp;rsquo;s public address is whatever IP the outside world sees the host as; that one varies.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The conversation between Frieda and Brutus that the outside world never saw.&lt;/strong&gt; This is the inter-container traffic on the bridge. Notice it crossed &lt;em&gt;no&lt;/em&gt; NAT, &lt;em&gt;no&lt;/em&gt; routing decision worth speaking of, &lt;em&gt;no&lt;/em&gt; outside interface. Two apartments, one corridor, Concierge passing parcels. Cheap, fast, invisible from outside. This is why container-to-container traffic on a bridge is so quick — the kernel basically never has to leave the lower layers.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The two ledgers being kept in lockstep.&lt;/strong&gt; That&amp;rsquo;s connection tracking — &lt;code&gt;conntrack&lt;/code&gt; — the kernel feature that remembers every active connection and what rewrites were applied to it, so reply traffic can be reversed perfectly even though the rewriting was applied asymmetrically. Without this, no NAT scheme on Earth would work. With it, the entire modern internet (where every home router does this, all day, for every device on your home network) functions invisibly.&lt;/p&gt;
&lt;h2 id="the-aha"&gt;The Aha
&lt;/h2&gt;
&lt;p&gt;When my friend at dinner heard &amp;ldquo;the internet,&amp;rdquo; she pictured, I think, a kind of pneumatic tube. You put a thing in one end, it comes out the other. Magic in the middle.&lt;/p&gt;
&lt;p&gt;The middle is not magic. The middle is a relay of small, dumb, identical actors, each doing one tiny job per parcel, none of them aware of the others, none of them aware of the contents. The &amp;ldquo;intelligence&amp;rdquo; lives at the two ends — Browser-Bot wrapping the parcel with the right addresses on each layer, and the receiving server unwrapping them and replying. The middle is just a sequence of helpful strangers passing notes.&lt;/p&gt;
&lt;p&gt;Inside the office building, the relay continues. Mama Kernel is more sophisticated than any one courier — she can rewrite addresses, route between corridors, keep ledgers — but she is still, fundamentally, a series of small mechanisms in sequence. Receive at the door. Strip the outer wrapper. Look at the address. Maybe rewrite it. Look up where it goes. Wrap it again. Send it out the right inner door. That&amp;rsquo;s a pipeline, not a wizard. You can hold the whole thing in your head once you stop expecting it to be one big magic step.&lt;/p&gt;
&lt;p&gt;This was the second click for me, after the switch one. The switch unlocked &lt;em&gt;what happens at one moment in time, in one place&lt;/em&gt;. The toaster story unlocked &lt;em&gt;what happens across an entire journey&lt;/em&gt;. They go together.&lt;/p&gt;
&lt;h2 id="what-i-still-dont-understand"&gt;What I Still Don&amp;rsquo;t Understand
&lt;/h2&gt;
&lt;p&gt;There&amp;rsquo;s a moment in the story I quietly waved past.&lt;/p&gt;
&lt;p&gt;When the parcel arrived at Frieda&amp;rsquo;s apartment, I said her &amp;ldquo;local kernel&amp;rdquo; peeled the wrappers, her own door slot received the courier sleeve, her own desk held the original packet. Frieda has, apparently, her own little internal world: her own front door, her own loopback to herself, her own apartment number, her own desk, her own everything.&lt;/p&gt;
&lt;p&gt;But Frieda isn&amp;rsquo;t a separate computer. Frieda is just a process running on the same host as Mama Kernel. There is only one kernel in this building. There is only one set of hardware.&lt;/p&gt;
&lt;p&gt;So how does Frieda&amp;rsquo;s apartment &lt;em&gt;exist&lt;/em&gt;? How does it have its own internal addresses, its own loopback, its own private experience of the corridor that doesn&amp;rsquo;t trample on Brutus&amp;rsquo;s right next door? How does the same kernel run multiple parallel &lt;em&gt;network worlds&lt;/em&gt; side by side, each believing it&amp;rsquo;s alone?&lt;/p&gt;
&lt;p&gt;The answer is the feature I waved at in post 1 without explaining. It&amp;rsquo;s called a &lt;strong&gt;network namespace&lt;/strong&gt;, and once it clicks, the entire reason containers feel like &amp;ldquo;their own little computers&amp;rdquo; snaps into focus. It&amp;rsquo;s also the foundation under VPNs, under sandboxing, under a lot of zero-trust networking — and under the bug I&amp;rsquo;m slowly explaining across this whole series.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;ll get there. I promise.&lt;/p&gt;
&lt;p&gt;See you next post — whatever rabbit hole I fall into first.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;(Written by Human, improved using AI where applicable.)&lt;/em&gt;&lt;/p&gt;</description></item><item><title>What a Switch Actually Does</title><link>https://www.shanechang.com/p/networking-journey-portainer-and-switches/</link><pubDate>Tue, 12 May 2026 00:00:00 +0000</pubDate><guid>https://www.shanechang.com/p/networking-journey-portainer-and-switches/</guid><description>&lt;img src="https://www.shanechang.com/p/networking-journey-portainer-and-switches/cover.webp" alt="Featured image of post What a Switch Actually Does" /&gt;&lt;p&gt;I was trying to do something I&amp;rsquo;d done a dozen times.&lt;/p&gt;
&lt;p&gt;Set up Portainer — the Docker management UI — on my laptop. Have it manage a remote server through a Cloudflare tunnel so nothing is exposed publicly. The remote machine runs a Portainer agent; the laptop runs the Portainer server; they talk over SSH wrapped in Cloudflare Access.&lt;/p&gt;
&lt;p&gt;I knew the moving parts. I&amp;rsquo;d touched all of them before. SSH tunnels, Docker, Portainer, Cloudflare — none of this was new.&lt;/p&gt;
&lt;p&gt;It didn&amp;rsquo;t work.&lt;/p&gt;
&lt;p&gt;The agent on the remote was alive. The UI on my laptop was alive. Both endpoints were reachable when I poked them from a terminal. But when I clicked &lt;strong&gt;Connect&lt;/strong&gt; in the Portainer UI, I got the cleanest, most damning error in all of networking:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Get &amp;#34;https://host.containers.internal:19001/ping&amp;#34;: dial tcp 10.88.0.1:19001: connect: connection refused
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&amp;ldquo;Connection refused.&amp;rdquo; The kind of error that means &lt;em&gt;something&lt;/em&gt; is rejecting your call. Not &amp;ldquo;no answer,&amp;rdquo; not &amp;ldquo;no route,&amp;rdquo; but a deliberate door slam.&lt;/p&gt;
&lt;p&gt;I stared at it for an embarrassing amount of time. Every link in the chain checked out individually. The agent was listening. The SSH tunnel had bound its port. &lt;code&gt;curl&lt;/code&gt; from the laptop to the tunnel worked. The Portainer container itself was healthy.&lt;/p&gt;
&lt;p&gt;Each piece was fine. The error said something about an interface I couldn&amp;rsquo;t even fully explain — &lt;code&gt;10.88.0.1&lt;/code&gt; — and a hostname I&amp;rsquo;d typed for years without really thinking about what it meant — &lt;code&gt;host.containers.internal&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That was the moment.&lt;/p&gt;
&lt;h2 id="the-honest-admission"&gt;The Honest Admission
&lt;/h2&gt;
&lt;p&gt;I&amp;rsquo;d been operating on memorized commands.&lt;/p&gt;
&lt;p&gt;I knew &lt;code&gt;ssh -L&lt;/code&gt; forwards a port. I knew &lt;code&gt;docker run -p 9001:9001&lt;/code&gt; &amp;ldquo;exposes&amp;rdquo; a port. I knew &lt;code&gt;127.0.0.1&lt;/code&gt; was &amp;ldquo;localhost.&amp;rdquo; I&amp;rsquo;d typed all of this hundreds of times and it had always just worked. And when it didn&amp;rsquo;t, I&amp;rsquo;d usually fix it by stack-overflowing my way to a different flag combination that did work, without ever understanding why.&lt;/p&gt;
&lt;p&gt;I learned the OSI model in college as seven boxes to memorize for the exam. Physical, data link, network, transport, session, presentation, application. I could&amp;rsquo;ve named them on a quiz. I could not have told you what the difference between a bridge and a router is. I&amp;rsquo;d worked professionally for years without filling that gap, because the abstractions sit on top of each other so cleanly that you can build whole careers on the top layer and pretend the bottom six aren&amp;rsquo;t there.&lt;/p&gt;
&lt;p&gt;But they are there. And as our company has scaled, the asks have started reaching deeper into the cake — reverse proxies, streaming, multi-region failover, zero-trust networking, &amp;ldquo;why does this work in dev but not in staging.&amp;rdquo; Every one of them pokes at exactly the gaps I&amp;rsquo;d been papering over.&lt;/p&gt;
&lt;p&gt;A &lt;code&gt;connection refused&lt;/code&gt; error from inside a container, against a port my own laptop was clearly serving, was just the latest version of the same gap. The bug wasn&amp;rsquo;t in the tools. The bug was in me.&lt;/p&gt;
&lt;p&gt;So I decided to actually learn it.&lt;/p&gt;
&lt;p&gt;This is post one of an open-ended series. The plan isn&amp;rsquo;t to write a textbook — the world has plenty of those, and I wouldn&amp;rsquo;t be the right author. The plan is to dig into the layer of the stack that&amp;rsquo;s directly behind whatever I&amp;rsquo;m currently confused by, until the confusion lifts, then move to the next layer. The Portainer bug is the spine. Every post returns to it.&lt;/p&gt;
&lt;p&gt;By the end of the series, the error message above will read like a sentence in plain English, not an incantation.&lt;/p&gt;
&lt;p&gt;This post is about the first thing I had to understand: &lt;strong&gt;what is actually happening at the lowest layer that&amp;rsquo;s relevant to my bug&lt;/strong&gt;. What is &lt;code&gt;10.88.0.1&lt;/code&gt;? Why is it different from &lt;code&gt;127.0.0.1&lt;/code&gt;? What is a &amp;ldquo;bridge,&amp;rdquo; really?&lt;/p&gt;
&lt;h2 id="wait-but-how-does-the-network-even-work-inside-my-laptop"&gt;&amp;ldquo;Wait, but how does the network even &lt;em&gt;work&lt;/em&gt; inside my laptop?&amp;rdquo;
&lt;/h2&gt;
&lt;p&gt;Here&amp;rsquo;s the part that hit me first.&lt;/p&gt;
&lt;p&gt;The agent on the remote was running on port &lt;code&gt;19001&lt;/code&gt;. My SSH tunnel forwarded my laptop&amp;rsquo;s port &lt;code&gt;19001&lt;/code&gt; to the remote&amp;rsquo;s port &lt;code&gt;19001&lt;/code&gt;. I could &lt;code&gt;curl https://127.0.0.1:19001/ping&lt;/code&gt; from my laptop terminal and get a clean reply.&lt;/p&gt;
&lt;p&gt;The Portainer Server, running inside a container &lt;em&gt;on the same laptop&lt;/em&gt;, could not reach the same address. It got &amp;ldquo;connection refused.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;Same machine. Same kernel. Same port number. Different result.&lt;/p&gt;
&lt;p&gt;How is that possible?&lt;/p&gt;
&lt;p&gt;The naive mental model — the one I&amp;rsquo;d been carrying — is that a computer has &lt;em&gt;a&lt;/em&gt; network. There&amp;rsquo;s &amp;ldquo;my machine,&amp;rdquo; and either something is reachable from it or it isn&amp;rsquo;t. If &lt;code&gt;curl&lt;/code&gt; works, anything else running on the same box should be able to do the same thing.&lt;/p&gt;
&lt;p&gt;That model is wrong, and it has been wrong since well before Docker existed. A modern Linux box is not a single network. It is a &lt;em&gt;network of networks&lt;/em&gt;, stitched together inside one kernel. Containers, VMs, VPNs, even ordinary SSH tunnels — they all rely on this fact. You just don&amp;rsquo;t have to think about it until something breaks weirdly.&lt;/p&gt;
&lt;p&gt;To make any sense of why my container couldn&amp;rsquo;t see what my terminal could, I had to start with the dumbest, most fundamental piece of the puzzle: the virtual switch sitting between my container and the rest of the machine.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m going to tell you about it as a story. Bear with me. There&amp;rsquo;s a small cast.&lt;/p&gt;
&lt;h2 id="a-small-cast"&gt;A Small Cast
&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Margaret.&lt;/strong&gt; A receptionist with infinite patience and a single notebook. She stands in the middle of a meeting room. Her entire job, all day, is to deliver paper notes between the people sitting at the table.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Alice, Bob, Carlos.&lt;/strong&gt; Three coworkers seated at the table. Each one wears a name badge clipped to their shirt. The badges are unique, machine-printed, and frankly weird-looking — they say things like &lt;code&gt;aa:42:81:9c:0e:33&lt;/code&gt;. Nobody&amp;rsquo;s ever asked why. They came that way.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Notebook.&lt;/strong&gt; Margaret&amp;rsquo;s notebook, in which she writes down — and only ever writes down — &lt;em&gt;who sits at which seat&lt;/em&gt;. Nobody helps her fill it in. She fills it in herself, just by watching.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Notes.&lt;/strong&gt; Folded paper. Each note has a name badge on the outside (who it&amp;rsquo;s for) and some content on the inside (what&amp;rsquo;s being said). Margaret only ever looks at the outside.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s the cast for the room. Now the story.&lt;/p&gt;
&lt;h2 id="the-conference-room"&gt;The Conference Room
&lt;/h2&gt;
&lt;p&gt;Alice has just started at the company. Today is her first day. She walks in, picks an empty seat — seat 3 — and sits down.&lt;/p&gt;
&lt;p&gt;Margaret notices nothing in particular. She has no idea who Alice is yet. The notebook says nothing about Alice.&lt;/p&gt;
&lt;p&gt;A few minutes later, Alice writes a quick &amp;ldquo;good morning&amp;rdquo; note to Carlos and hands it to Margaret. Margaret takes the note, reads the destination — Carlos&amp;rsquo;s name badge — and thinks: &amp;ldquo;Carlos? Carlos. Seat 7.&amp;rdquo; She delivers it to seat 7.&lt;/p&gt;
&lt;p&gt;But before she walks off, Margaret also does something almost imperceptible. She glances at the &lt;em&gt;sender&amp;rsquo;s&lt;/em&gt; name on the outside of the envelope — Alice&amp;rsquo;s badge — and the &lt;em&gt;seat&lt;/em&gt; she just picked the note up from. She writes a line in her notebook:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;Alice (badge: aa:42:81:9c:0e:33) — seat 3.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Now she knows.&lt;/p&gt;
&lt;p&gt;Ten minutes later, Bob (seat 5) wants to send Alice a note. He writes it, addresses it to Alice&amp;rsquo;s badge, and hands it to Margaret. Margaret flips open her notebook, looks up Alice, sees &amp;ldquo;seat 3,&amp;rdquo; and walks the note over. Done.&lt;/p&gt;
&lt;p&gt;This is the calm, well-run version of Margaret&amp;rsquo;s day. Everyone she&amp;rsquo;s seen, she knows. Looking up a seat is a one-second flip of a page.&lt;/p&gt;
&lt;p&gt;But what about the moment &lt;em&gt;before&lt;/em&gt; she knew Alice existed? What if Bob had wanted to send Alice a note five minutes into Alice&amp;rsquo;s first day, while she was still settling in, before Alice had sent so much as a single note Margaret could learn from?&lt;/p&gt;
&lt;p&gt;Margaret would flip open the notebook. Alice&amp;rsquo;s name wouldn&amp;rsquo;t be there. She&amp;rsquo;d shrug, walk around the room, and hand a copy of the note to &lt;em&gt;everyone except Bob&lt;/em&gt;. The way she sees it: I don&amp;rsquo;t know who Alice is, so the simplest thing is to give everyone a copy and let them sort it out. Whichever one is Alice will read her copy and react; everyone else will glance at the name on the outside, see it&amp;rsquo;s not theirs, and drop the copy in the bin.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s it. That&amp;rsquo;s Margaret&amp;rsquo;s &lt;em&gt;entire&lt;/em&gt; job. Two behaviours.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;If she knows the seat:&lt;/strong&gt; look up, deliver. Fast, quiet, no fuss.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;If she doesn&amp;rsquo;t know the seat:&lt;/strong&gt; hand a copy to everyone. Crude but effective.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And every note she sees — whether it&amp;rsquo;s the one she&amp;rsquo;s looking up &lt;em&gt;or&lt;/em&gt; the one she&amp;rsquo;s about to flood — she also passively learns from. &lt;em&gt;&amp;ldquo;Note came up from seat 3, sender badge aa:42:81:9c:0e:33&amp;rdquo;&lt;/em&gt; — into the notebook it goes. Plug a new person in, have them send one note, and Margaret knows where they are. Forever. She&amp;rsquo;s never wrong, because she only writes down what she&amp;rsquo;s actually seen with her own eyes.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s the whole conference room. Two mechanisms, one self-filling notebook, zero knowledge of anything above her own little job.&lt;/p&gt;
&lt;h2 id="what-margaret-was-in-computer-terms"&gt;What Margaret Was, In Computer Terms
&lt;/h2&gt;
&lt;p&gt;Time to break the spell, briefly, and say who Margaret really is.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Margaret is a switch.&lt;/strong&gt; In software form she&amp;rsquo;s also called a &lt;strong&gt;bridge&lt;/strong&gt;, but the algorithm is identical. Hardware switches sit in rack-mounted metal boxes in data centres. Software bridges sit as data structures in your laptop&amp;rsquo;s kernel. They behave the same.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The seats&lt;/strong&gt; are &lt;strong&gt;ports&lt;/strong&gt; on the switch — the actual jacks where Ethernet cables plug in. &lt;em&gt;Not&lt;/em&gt; the TCP ports you might be thinking of. Different concept, same word. Sorry.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The name badges&lt;/strong&gt; are &lt;strong&gt;MAC addresses&lt;/strong&gt;. Every network card on Earth has one, burned in at the factory. They look exactly as weird as Margaret&amp;rsquo;s: six pairs of hex characters separated by colons.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Margaret&amp;rsquo;s notebook&lt;/strong&gt; is the switch&amp;rsquo;s &lt;strong&gt;MAC address table&lt;/strong&gt;. The mapping of &amp;ldquo;this badge lives at this seat.&amp;rdquo;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Behaviour 1 (lookup-and-deliver)&lt;/strong&gt; is called &lt;strong&gt;forwarding&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Behaviour 2 (hand-to-everyone)&lt;/strong&gt; is called &lt;strong&gt;flooding&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The trick where Margaret writes down what she observes&lt;/strong&gt; is called &lt;strong&gt;MAC learning&lt;/strong&gt;, and it is genuinely one of the most elegant designs in the whole stack. The switch self-configures from passively watching traffic. No DHCP, no config file, no setup wizard. Plug a device in. Have it transmit one frame. The switch knows where it is.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That last point matters. Margaret doesn&amp;rsquo;t read the notes. Margaret only looks at the outside. She has no idea what protocol the senders are using, what language they&amp;rsquo;re writing in, what they&amp;rsquo;re talking about. She doesn&amp;rsquo;t need to. Forward or flood. Repeat.&lt;/p&gt;
&lt;p&gt;This is the reason physical switches can forward millions of frames per second in dedicated silicon. There&amp;rsquo;s almost nothing to compute. Read source badge, update notebook. Read destination badge, look up or flood. Done.&lt;/p&gt;
&lt;h2 id="bob-asks-for-someone-he-doesnt-know"&gt;Bob Asks for Someone He Doesn&amp;rsquo;t Know
&lt;/h2&gt;
&lt;p&gt;Now the obvious next question. The one that snagged me.&lt;/p&gt;
&lt;p&gt;Okay — Margaret can lookup-or-flood. Fine. But how does Bob know what badge to write on the outside of his envelope in the first place? When I type an IP address into a browser, my computer eventually has to know which physical network card owns that IP. Who answers &lt;em&gt;that&lt;/em&gt; question?&lt;/p&gt;
&lt;p&gt;Back to the room.&lt;/p&gt;
&lt;p&gt;Today, Bob has a problem. He&amp;rsquo;s been asked to send a note to &amp;ldquo;the person who handles toaster orders.&amp;rdquo; He has &lt;em&gt;no idea&lt;/em&gt; which of his coworkers that is. He doesn&amp;rsquo;t know their name badge. He doesn&amp;rsquo;t know their seat.&lt;/p&gt;
&lt;p&gt;Margaret can&amp;rsquo;t help directly. Her notebook maps badges to seats; it doesn&amp;rsquo;t tell you what people &lt;em&gt;do&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;But Bob knows a trick.&lt;/p&gt;
&lt;p&gt;He writes his note. The note inside says, &amp;ldquo;Hi — I&amp;rsquo;m Bob, badge &lt;code&gt;bb:11:22:33:44:55&lt;/code&gt;, seat 5. Whoever handles toaster orders, please write back and tell me your badge.&amp;rdquo; Then on the &lt;em&gt;outside&lt;/em&gt; of the envelope, in the destination field, instead of writing a specific person&amp;rsquo;s badge, he writes:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;code&gt;FF:FF:FF:FF:FF:FF&lt;/code&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This is a special, reserved name badge. &lt;em&gt;Nobody actually wears it.&lt;/em&gt; It&amp;rsquo;s a magic phrase. The whole room knows, by convention, that this badge means &lt;strong&gt;&amp;ldquo;everyone.&amp;rdquo;&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;He hands the envelope to Margaret.&lt;/p&gt;
&lt;p&gt;Margaret looks at the destination: &lt;code&gt;FF:FF:FF:FF:FF:FF&lt;/code&gt;. She flips open the notebook. It&amp;rsquo;s not there — of course it isn&amp;rsquo;t, nobody wears that badge. So she does the only thing she ever does when a destination isn&amp;rsquo;t in her notebook: she walks around the room and hands a copy to every seat except Bob&amp;rsquo;s.&lt;/p&gt;
&lt;p&gt;Notice what just happened. Margaret didn&amp;rsquo;t do anything special. She didn&amp;rsquo;t &amp;ldquo;recognise&amp;rdquo; Bob&amp;rsquo;s question or &amp;ldquo;decide&amp;rdquo; to broadcast. She got a destination she couldn&amp;rsquo;t find in the notebook and fell into her flood behaviour. The smart move was on the &lt;em&gt;sender&amp;rsquo;s&lt;/em&gt; side. Bob picked the magic badge &lt;em&gt;because&lt;/em&gt; he wanted Margaret to flood for him.&lt;/p&gt;
&lt;p&gt;Everyone gets a copy. Most people glance at the inside, see &amp;ldquo;toaster orders&amp;rdquo; doesn&amp;rsquo;t apply to them, and drop their copy in the bin. But Alice — who, it turns out, is in charge of toaster orders — reads her copy carefully. Then she writes back to Bob, this time with the destination set to Bob&amp;rsquo;s actual badge, content: &amp;ldquo;Hi Bob, I handle toasters, my badge is &lt;code&gt;aa:42:81:9c:0e:33&lt;/code&gt;.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;Margaret, in delivering Alice&amp;rsquo;s reply back to seat 5, &lt;em&gt;also&lt;/em&gt; passively writes Alice into her notebook (if she hadn&amp;rsquo;t already). Bob now knows Alice&amp;rsquo;s badge. Margaret now knows Alice&amp;rsquo;s seat. Everyone has learned what they need to know from a single exchange.&lt;/p&gt;
&lt;p&gt;In computer terms: that exchange is called &lt;strong&gt;ARP&lt;/strong&gt; — Address Resolution Protocol. The magic badge &lt;code&gt;FF:FF:FF:FF:FF:FF&lt;/code&gt; is called the &lt;strong&gt;broadcast MAC address&lt;/strong&gt;. The whole protocol is just: send a question to everyone using the magic badge, the right device replies, you remember its real badge, and now you can talk directly.&lt;/p&gt;
&lt;p&gt;This was the bit that clicked hardest for me. ARP isn&amp;rsquo;t a &lt;em&gt;third&lt;/em&gt; behaviour on top of forward-and-flood. It&amp;rsquo;s a clever &lt;em&gt;use&lt;/em&gt; of the flood behaviour, triggered by the sender choosing the magic destination. The switch is along for the ride. It can&amp;rsquo;t tell ARP apart from regular traffic, and it doesn&amp;rsquo;t need to.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Trigger&lt;/th&gt;
&lt;th&gt;Why it floods&lt;/th&gt;
&lt;th&gt;Who decided&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unknown destination badge&lt;/td&gt;
&lt;td&gt;Switch has no entry, falls back&lt;/td&gt;
&lt;td&gt;The switch (forced by ignorance)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Destination badge is &lt;code&gt;FF:FF:FF:FF:FF:FF&lt;/code&gt; (broadcast)&lt;/td&gt;
&lt;td&gt;Sender wrote &amp;ldquo;everyone&amp;rdquo; on envelope&lt;/td&gt;
&lt;td&gt;The sender (intentional)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;img src="https://www.shanechang.com/p/networking-journey-portainer-and-switches/how_arp_works_through_a_switch.png"
width="2172"
height="724"
srcset="https://www.shanechang.com/p/networking-journey-portainer-and-switches/how_arp_works_through_a_switch_hu_8f3c2f028502d60f.png 480w, https://www.shanechang.com/p/networking-journey-portainer-and-switches/how_arp_works_through_a_switch_hu_d8f8c70f4f78a8ee.png 1024w"
loading="lazy"
alt="How an ARP request travels through a switch: the sender broadcasts to FF:FF:FF:FF:FF:FF, the switch floods every port, and only the IP owner replies — letting the switch learn that MAC’s seat on the way back."
class="gallery-image"
data-flex-grow="300"
data-flex-basis="720px"
&gt;
&lt;/p&gt;
&lt;p&gt;(One small footnote that delighted me when I learned it: &lt;code&gt;FF:FF:FF:FF:FF:FF&lt;/code&gt; being &amp;ldquo;all 1s&amp;rdquo; isn&amp;rsquo;t arbitrary. The lowest bit of the first byte of a MAC address is reserved to mean &amp;ldquo;this is a group address.&amp;rdquo; Hardware can check &amp;ldquo;is this a broadcast?&amp;rdquo; with a single-bit test on one byte — cheaper than a notebook lookup. Hardware-friendliness was baked into the protocol from day one.)&lt;/p&gt;
&lt;h2 id="the-room-goes-virtual"&gt;The Room Goes Virtual
&lt;/h2&gt;
&lt;p&gt;Everything I just described is the world of a &lt;em&gt;physical&lt;/em&gt; switch — a metal box in a rack, with Ethernet jacks on the front. Margaret is sitting in an actual room, with actual humans, passing actual paper.&lt;/p&gt;
&lt;p&gt;The switch on my laptop is virtual. Linux runs it. The kernel calls these things &lt;strong&gt;bridges&lt;/strong&gt;. Docker&amp;rsquo;s is named &lt;code&gt;docker0&lt;/code&gt;; Podman&amp;rsquo;s is named &lt;code&gt;podman0&lt;/code&gt;. Behaviourally, it is identical to the physical box: same notebook, same two mechanisms, same broadcast handling, same total ignorance of anything above its own little job. The difference is that Margaret has moved into the kernel, and the room around her is now made of data structures in memory rather than transistors and copper.&lt;/p&gt;
&lt;p&gt;When a container starts, the kernel creates a &lt;strong&gt;virtual Ethernet cable&lt;/strong&gt; — a &lt;code&gt;veth&lt;/code&gt; pair. Think of it as a literal cable with two ends. One end gets moved into the container, where it shows up as &lt;code&gt;eth0&lt;/code&gt;. The other end gets plugged into a seat in Margaret&amp;rsquo;s room. The kernel guarantees: whatever goes in one end comes out the other. That&amp;rsquo;s the entire abstraction.&lt;/p&gt;
&lt;p&gt;And here is the part that, for me, was the unlock: &lt;strong&gt;the host is also seated in Margaret&amp;rsquo;s room.&lt;/strong&gt; When Margaret&amp;rsquo;s room is first set up, the kernel gives the host its own seat at the table — its own badge, its own IP. Margaret does not know or care which seat belongs to the host and which belong to containers. They&amp;rsquo;re all just people with name badges, and she passes notes between them by the only rules she has.&lt;/p&gt;
&lt;p&gt;Docker and Podman, &lt;em&gt;at this layer&lt;/em&gt;, are doing three boring things:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Ask the kernel to create the room (the bridge).&lt;/li&gt;
&lt;li&gt;Plug containers in and out via &lt;code&gt;veth&lt;/code&gt; pairs as they start and stop.&lt;/li&gt;
&lt;li&gt;Make sure the host gets its own seat at the table.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The kernel does the actual switching. Docker and Podman are just running the room.&lt;/p&gt;
&lt;p&gt;And here is where my bug starts to make sense: &lt;strong&gt;&lt;code&gt;10.88.0.1&lt;/code&gt; is the host&amp;rsquo;s seat at Margaret&amp;rsquo;s table in Podman&amp;rsquo;s room.&lt;/strong&gt; It&amp;rsquo;s the IP of the host&amp;rsquo;s port on &lt;code&gt;podman0&lt;/code&gt;. From inside a container, that IP is &amp;ldquo;the host, as seen from inside this room.&amp;rdquo; That is what &lt;code&gt;host.containers.internal&lt;/code&gt; is a friendly DNS name for. It&amp;rsquo;s not a magic concept — it&amp;rsquo;s the host&amp;rsquo;s seat at this particular meeting room. (Docker is the same idea with different numbers: &lt;code&gt;172.17.0.1&lt;/code&gt; is where the host sits in &lt;code&gt;docker0&lt;/code&gt;&amp;rsquo;s room.)&lt;/p&gt;
&lt;h2 id="but-the-building-has-many-rooms"&gt;But the Building Has Many Rooms
&lt;/h2&gt;
&lt;p&gt;If the host is sitting in Margaret&amp;rsquo;s room with an IP of &lt;code&gt;10.88.0.1&lt;/code&gt;, you might be wondering: what about &lt;code&gt;127.0.0.1&lt;/code&gt;? What about good old localhost?&lt;/p&gt;
&lt;p&gt;Localhost is &lt;em&gt;a different room&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Imagine the office building has, somewhere down the hall, a tiny private room where each company runs its own internal mail circle. The room only has one seat in it — yours — and a separate receptionist, &lt;strong&gt;Lorraine&lt;/strong&gt;. Lorraine&amp;rsquo;s whole job is even simpler than Margaret&amp;rsquo;s: when you hand her a note, she walks two steps and hands it back to you. That&amp;rsquo;s it. There&amp;rsquo;s nobody else in the room. Notes you write to yourself in Lorraine&amp;rsquo;s room never leave.&lt;/p&gt;
&lt;p&gt;Lorraine is the &lt;strong&gt;loopback interface&lt;/strong&gt; — &lt;code&gt;lo&lt;/code&gt;, address &lt;code&gt;127.0.0.1&lt;/code&gt;. Every Linux box has one. It&amp;rsquo;s the room where a program talks to itself.&lt;/p&gt;
&lt;p&gt;The host has a seat in &lt;em&gt;both&lt;/em&gt; rooms. The host sits with Margaret (&lt;code&gt;10.88.0.1&lt;/code&gt;) and also sits with Lorraine (&lt;code&gt;127.0.0.1&lt;/code&gt;). Different rooms, different receptionists, different addresses.&lt;/p&gt;
&lt;p&gt;Here&amp;rsquo;s the part I&amp;rsquo;d never properly understood. When a program on the host opens a network listener — call it a doorman waiting to accept connections — it has to decide &lt;em&gt;which room&amp;rsquo;s door to put the doorman in front of&lt;/em&gt;. Bind your doorman to &lt;code&gt;127.0.0.1&lt;/code&gt; and the doorman is standing in Lorraine&amp;rsquo;s room; anyone wanting to talk to him has to be sitting in Lorraine&amp;rsquo;s room too. Bind your doorman to &lt;code&gt;10.88.0.1&lt;/code&gt; and he&amp;rsquo;s standing in Margaret&amp;rsquo;s room. Bind him to &lt;code&gt;0.0.0.0&lt;/code&gt; (&amp;ldquo;any room&amp;rdquo;) and he stands in every room the host has a seat in.&lt;/p&gt;
&lt;p&gt;This is &lt;em&gt;exactly&lt;/em&gt; where my bug lived.&lt;/p&gt;
&lt;p&gt;When my SSH tunnel set up its listener — &lt;em&gt;&amp;ldquo;please accept incoming connections on port 19001 and forward them out to the remote server&amp;rdquo;&lt;/em&gt; — it bound the doorman to &lt;code&gt;127.0.0.1&lt;/code&gt;. The doorman set up shop in Lorraine&amp;rsquo;s room.&lt;/p&gt;
&lt;p&gt;When I, on the host, ran &lt;code&gt;curl https://127.0.0.1:19001/ping&lt;/code&gt; from a terminal, I was sitting in Lorraine&amp;rsquo;s room asking for the door to be answered. The doorman was right there. He answered. &lt;code&gt;curl&lt;/code&gt; worked.&lt;/p&gt;
&lt;p&gt;But when the Portainer container — sitting at the table in Margaret&amp;rsquo;s room — sent its request to the host&amp;rsquo;s interface on the bridge (&lt;code&gt;10.88.0.1:19001&lt;/code&gt;), the request arrived through Margaret&amp;rsquo;s room. The container knocked on a door in Margaret&amp;rsquo;s room. &lt;em&gt;The doorman wasn&amp;rsquo;t there.&lt;/em&gt; He was over in Lorraine&amp;rsquo;s room. The host had no listener on this interface for this port. &lt;em&gt;Connection refused.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The fix, once I could see it, was idiotic in its smallness. Bind the SSH tunnel&amp;rsquo;s listener to &lt;code&gt;10.88.0.1&lt;/code&gt; instead. Or &lt;code&gt;0.0.0.0&lt;/code&gt;. Anything that put the doorman in the room where the container&amp;rsquo;s knock would actually land.&lt;/p&gt;
&lt;p&gt;But getting to &amp;ldquo;idiotic in its smallness&amp;rdquo; required me to know there were multiple rooms at all. Which required me to know what a bridge was. Which required me to know that Margaret only does two things. Which is why this is post one.&lt;/p&gt;
&lt;h2 id="so-did-it-all-make-sense"&gt;So, Did It All Make Sense?
&lt;/h2&gt;
&lt;p&gt;It made &lt;em&gt;more&lt;/em&gt; sense. Not all sense.&lt;/p&gt;
&lt;p&gt;I now understood why &lt;code&gt;10.88.0.1&lt;/code&gt; mattered and why it isn&amp;rsquo;t the same as &lt;code&gt;127.0.0.1&lt;/code&gt;. I could see Margaret&amp;rsquo;s room in my head — a virtual conference room with my container at one seat and the host at another, Margaret passing notes by name badge, doing forward-or-flood and nothing else. I could see why a listener bound to one interface — a doorman in one room — was invisible to traffic arriving from another. I could see why ARP wasn&amp;rsquo;t magic — just Bob using the magic &amp;ldquo;everyone&amp;rdquo; badge to ask a question, and the switch faithfully flooding because that&amp;rsquo;s all it knows how to do.&lt;/p&gt;
&lt;p&gt;But a much weirder question had taken its place. The whole story above assumes that &lt;em&gt;a container has its own network stack to plug into Margaret&amp;rsquo;s room&lt;/em&gt;. The container is, supposedly, just a process running on my laptop&amp;rsquo;s kernel. So how does a process get its own network card? Its own IP? Its own seat in Lorraine&amp;rsquo;s private little room that isn&amp;rsquo;t the host&amp;rsquo;s seat in Lorraine&amp;rsquo;s room?&lt;/p&gt;
&lt;p&gt;And come to think of it — when I&amp;rsquo;d been confidently saying &amp;ldquo;containers are isolated,&amp;rdquo; what did I actually mean by isolated? Isolated from what? Using what mechanism? Where in the kernel does that isolation live?&lt;/p&gt;
&lt;p&gt;I had filled in one gap and immediately found a bigger one underneath it.&lt;/p&gt;
&lt;p&gt;Network namespaces are coming — the kernel feature that lets a single Linux machine pretend to be many independent networks, all running side by side, all believing they have their own &lt;code&gt;127.0.0.1&lt;/code&gt;. It&amp;rsquo;s the foundation of containers, VPNs, and a surprising amount of modern infrastructure — and once it clicks, the &lt;code&gt;veth&lt;/code&gt; pair I waved at above suddenly stops being a metaphor and starts being a piece of obvious plumbing.&lt;/p&gt;
&lt;p&gt;See you next post.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;(Written by Human, improved using AI where applicable.)&lt;/em&gt;&lt;/p&gt;</description></item></channel></rss>