# HTTP/2 Under the Hood: Frames, Streams, and Why Binary Wins It's been over two months since my last post. I started my internship, and between that and the side projects I'm building, the time I have for this series is a lot smaller than it was. I'm still working through the series, just slower than before. The last post ended with a promise to go deeper on HTTP/2. Before multiplexing and prioritization make sense, I had to answer a more basic question: what does an HTTP/2 message actually look like on the wire? That led to two things I learned back to back, so this post covers both: **frames** and **HPACK**. --- ## 1. Why This Topic Matters In the last post, I said HTTP/2 gives every request a stream ID, and that's what makes multiplexing possible. But I left it abstract. Where does the stream ID live? Who reads it? And if everything is split up, how does the server know what each piece is? The answer is the **frame**. Once you understand it, a lot of HTTP/2 stops feeling like separate features, because they are all built on this one small structure. Frames also raise a second problem. If every request is a HEADERS frame, then the same headers (host, user agent, cookies, accept) are sent again and again on a connection. HTTP/2 fixes the request format, but it would still waste bytes on repeated text. That's what **HPACK** solves. ## 2. Source Material Playlist: Chai aur Code — Computer Networking Series Video: HTTP/1.1 vs HTTP/2 [RFC 9113 — HTTP/2](https://www.rfc-editor.org/rfc/rfc9113) [RFC 7541 — HPACK](https://www.rfc-editor.org/rfc/rfc7541) [HPACK static table (RFC 7541, Appendix A)](https://rfcinfo.com/rfc-7541/appendix-a/) [Huffman table (background reading)](https://www.sciencedirect.com/topics/engineering/huffman-table) [HTTP 2 frames in depth](https://youtu.be/dX0DK0zt6WU) [HPACK explained in depth](https://youtu.be/M5ecS9oPRtk) ## 3. What I Learned HTTP/1.1 is a text protocol. A server reading a POST request goes through it line by line. The first line holds the method, path and version. Then come the headers, one per line, each ending in `\r\n`. A blank line marks the end of the headers, and the body starts right after it. The server already knows the body's size from `Content-Length`, so it reads exactly that many bytes. It works, but the server is always scanning text for line breaks, and the same plain-text headers are sent again on every request. HTTP/2 changes the format, not the meaning. Every message is split into **frames**, and every frame starts with a fixed 9-byte header that says how long the payload is, what type of frame it is, and which stream it belongs to. Since that header never changes shape, the server doesn't search for anything. It reads fixed positions and jumps straight to the payload. A request is made of two main frame types: a **HEADERS frame** (everything except the body) and a **DATA frame** (the body). A HEADERS frame always comes first. The stream ID in each frame is what ties the pieces together. Frames from different requests can be mixed on one connection, and the server uses the stream ID to rebuild each request. The response comes back the same way, in frames carrying the same stream ID as the request. The payload of a HEADERS frame isn't plain text either. It's compressed with **HPACK**, which uses three tools together. The **static table** is a fixed list of 61 common headers. The **dynamic table** is built during the connection from headers that were actually sent. **Huffman coding** shrinks the leftover text. The idea underneath is simple: if both sides keep the same table, you can send a number instead of the text. ## 4. Key Concepts **Binary protocol** — data on the wire is raw bytes with a fixed layout, not text meant for humans. **Frame** — the smallest unit of HTTP/2: a 9-byte header followed by a payload. **HEADERS frame (type `0x01`)** — carries the request or response headers, HPACK encoded. **DATA frame (type `0x00`)** — carries the body. **Length (24 bits)** — payload size, so the server knows how many bytes to read next. **Flags (8 bits)** — small extra instructions, like `END_STREAM` (last frame of this stream) or `END_HEADERS` (headers are complete). **Stream ID (31 bits)** — which request/response this frame belongs to. **HPACK** — HTTP/2's header compression, applied only to the HEADERS frame payload. **Static table** — 61 predefined header entries, the same on every client and server, never changing. **Dynamic table** — a table built during the connection from headers actually sent. Its entries start at index 62. **Index** — the number of an entry in the tables. 1 to 61 is static, 62 and up is dynamic. **Literal** — a header sent as actual text, because it isn't fully available in a table. **Huffman coding** — optional string compression using a fixed table: common characters get shorter codes. **H bit** — one bit in front of a string's length. `1` means the string is Huffman coded. ## 5. How It Works ### The frame header Every frame starts with this fixed layout: ``` +-----------------------------------------------+ | Length (24 bits) | +---------------+---------------+---------------+ | Type (8) | Flags (8) | +-+-------------+-------------------------------+ |R| Stream Identifier (31 bits) | +-+---------------------------------------------+ | Frame Payload (...) | +-----------------------------------------------+ ``` 24 + 8 + 8 + 1 + 31 = 72 bits, which is **9 bytes**. The R bit is reserved and always 0. ### How the server reads one frame ``` 1. Read 24 bits → Length: the payload is N bytes 2. Read 8 bits → Type: 0x01 = HEADERS, 0x00 = DATA 3. Read 8 bits → Flags 4. Read 1 bit → Reserved (ignored) 5. Read 31 bits → Stream ID: which request is this? 6. Read N bytes → Payload, then move on to the next frame ``` If the type is HEADERS, the server knows the payload is HPACK-encoded and decodes it. If it's DATA, the payload is the body as is. No searching, no guessing. A real DATA frame carrying 5 bytes on stream 1, and it's the last frame of that stream: ``` 00 00 05 00 01 00 00 00 01 [5 bytes of body] Length=5 DATA END_STREAM Stream ID=1 ``` Compare that with HTTP/1.1, where finding the end of the headers means walking through text until you hit a blank line. ### Three requests, one connection Say the client sends three requests: | # | Request | Frames | |---|---|---| | 1 | POST with a body | HEADERS + DATA | | 2 | GET | HEADERS only | | 3 | GET for an HTML file | HEADERS only | Requests 2 and 3 have no body, so each is a single frame. All frames are sent independently and can be mixed on the same TCP connection: ``` [HEADERS s1] [HEADERS s3] [DATA s1] [HEADERS s5] ``` The server uses the stream ID to put them back together. Every frame of request 1 carries stream ID 1, so it collects the HEADERS and DATA frames of stream 1 and rebuilds the full request, even though other requests' frames arrived in between. The response goes the other way with the **same stream ID**: ``` Request → HEADERS (s1) + DATA (s1) Response ← HEADERS (s1: status, headers) + DATA (s1: body) ``` ### Where HPACK fits ``` HTTP/2 frame ├── 9-byte frame header (Length, Type = HEADERS, Flags, Stream ID) ← not compressed └── payload ← HPACK-encoded header block ``` HPACK only touches headers, in both requests and responses. DATA frames aren't compressed by it, and the frame header stays uncompressed so the server can read it in a fixed format. ### HPACK tool 1: the static table A fixed list of 61 common headers. A few of them: | Index | Name | Value | |---|---|---| | 1 | `:authority` | (empty) | | 2 | `:method` | `GET` | | 3 | `:method` | `POST` | | 4 | `:path` | `/` | | 5 | `:path` | `/index.html` | | 6 | `:scheme` | `http` | | 7 | `:scheme` | `https` | | 8 | `:status` | `200` | | 24 | `cache-control` | (empty) | | 58 | `user-agent` | (empty) | There are two kinds of entries: - **Name and value fixed** (`:method: GET`): the whole header becomes a single index. Nothing else is sent. - **Name only, value empty** (`:authority`, `user-agent`): the name is known, but the value changes every time, so the value still has to be sent as a literal. ### HPACK tool 2: Huffman coding For those literal values, HPACK can compress the text with Huffman coding. The table is **fixed and predefined**. It isn't learned from earlier requests. Common characters get short codes and rare ones get long codes. For example, `www.example.com` is 15 bytes as text and 12 bytes after Huffman. It's optional per string. The H bit in front of the length says whether the string is Huffman coded. Huffman output rarely ends on a byte boundary, so it's padded with 1-bits up to the next byte. Those 1s are the start of the special "end of string" code (all 1s), so the padding can never be mistaken for a real character. ### HPACK tool 3: the dynamic table The static table can't hold things like `www.example.com`, because that's different for every site. So a second table is built during the connection. - It starts **empty** when the connection starts. - When a new header is sent in full, it can be added to the table with index **62**. Older entries shift up by one. - The next time the same header appears, only the index is sent. - It has a size limit. When it's full, the **oldest entries are dropped first**. It's a FIFO queue. - Client and server each keep their own copy. They never send the table to each other. Both build the same table by reading the same header blocks in the same order. - It only lives for one connection. A new connection starts with an empty table. ### How the receiver knows what it's reading The first bits of each entry say what kind it is: | First bits | Type | Meaning | |---|---|---| | `1xxxxxxx` | Indexed header field | Whole header is in a table. Remaining 7 bits = index | | `01xxxxxx` | Literal with incremental indexing | Send it and add it to the dynamic table. Remaining 6 bits = name index | | `0000xxxx` | Literal without indexing | Send it, don't add it (value changes every request) | | `0001xxxx` | Literal never indexed | Send it, never add it, and proxies must not add it either (sensitive values) | A literal whose name is already in a table: ``` 01 + name index (6 bits) H bit + value length (7 bits) value string ``` A literal whose name is new: ``` 01 000000 (index 0 = new name) H bit + name length (7 bits) name string H bit + value length (7 bits) value string ``` The "never indexed" type exists for security. If secret values like tokens went into a shared compression table, an attacker could guess them by watching how the compressed size changes. `0001` keeps sensitive headers out of every table. `0000` is only about not wasting table space. ### Worked example This is the example from RFC 7541 (Appendix C), so every byte can be checked against the spec. The client sends two requests to `www.example.com`. **Request 1:** `GET http://www.example.com/` ``` 82 1 0000010 → index 2 → :method: GET 86 1 0000110 → index 6 → :scheme: http 84 1 0000100 → index 4 → :path: / 41 01 000001 → literal, add to table, name = index 1 (:authority) 8c 1 0001100 → H = 1 (Huffman), length = 12 bytes f1e3c2e5f23a6ba0ab90f4ff → Huffman("www.example.com") ``` That's **17 bytes**, against about 60 characters of plain text. The dynamic table now holds: ``` [62] :authority: www.example.com ``` **Request 2:** the same request plus `cache-control: no-cache` ``` 82 → :method: GET (static, index 2) 86 → :scheme: http (static, index 6) 84 → :path: / (static, index 4) be 1 0111110 → index 62 → :authority: www.example.com (dynamic table) 58 01 011000 → literal, add to table, name = index 24 (cache-control) 86 1 0000110 → H = 1, length = 6 bytes a8eb10649cbf → Huffman("no-cache") ``` That's **12 bytes**, for a request with more headers than the first one. `:authority` cost 14 bytes in request 1 (`41`, `8c` and 12 bytes of Huffman) and 1 byte in request 2 (`be`). The dynamic table now holds: ``` [62] cache-control: no-cache [63] :authority: www.example.com ← shifted from 62 to 63 when the new entry was added ``` ## 6. Things That Confused Me **The frame header size.** My first notes said "4 bytes" and "9 bits". The header is **9 bytes (72 bits)**. I mixed up bits and bytes. Adding the fields (24 + 8 + 8 + 1 + 31 = 72) is the easiest way to remember it. **Where did the request line go?** HTTP/1.1 has `POST /api HTTP/1.1` as the first line. HTTP/2 has no text line. That information goes into the HEADERS frame as pseudo-headers (`:method`, `:path`, `:scheme`, `:authority`), inside the HPACK payload. Same information, different format. **"Independent" doesn't mean "unordered."** Frames from different streams can be mixed freely, but frames inside the same stream still arrive in order. The stream ID is what lets the server keep streams apart. **One request isn't always two frames.** A large body can be split across several DATA frames, because there's a maximum frame size (16,384 bytes by default). The idea stays the same: the Length field says how much to read. **Static table doesn't mean "the whole header is in it."** Some entries have a fixed value (`:method: GET`), but some only have a name (`:authority`). For those, the name is an index and the value is still a literal. **Huffman isn't built from earlier requests.** I first thought it worked like the dynamic table, learning from history. It doesn't. Its table is fixed. The dynamic table is the part that learns. **FIFO, not FILO.** I first wrote that the dynamic table works like a stack. It's a queue: new entries go in at the front, and the oldest ones are removed first when it's full. **The index bits.** I wrote `1 000010` for index 2, but there are 7 bits after the leading 1: `1 0000010`, which is `0x82`. Count the bits when you decode by hand. **How do both sides end up with the same table?** They never exchange it. Each side applies the same instructions to its own table, in the same order, so the tables stay identical. ## 7. My Explanation in Simple Words HTTP/1.1 is a long letter. To find where the address ends and the message begins, you read line by line until you hit a blank line. HTTP/2 is more like parcels with a printed label in the same spot on every box. The label says how heavy the box is, what type it is, and which order it belongs to. The warehouse worker never opens the box to find that out. They read the label, put the box on the right pile, and move on. The stream ID is the order number on the label: boxes from many orders can ride the same conveyor belt, and each one still ends up in the right pile. HPACK is about the paperwork inside each box. Two people on a long phone call keep needing the same long phrases. Before the call, they get the same printed phrasebook with 61 common phrases, each with a number. That's the static table. During the call, they also keep a shared notebook. The first time you say something new in full, both of you write it down with the next number, and from then on you just say the number. That's the dynamic table. The notebook has limited pages, so when it's full the oldest line gets crossed out first. When you do have to say something in full, you use shorthand where common letters take less space, like Morse code gives `e` a single dot. That's Huffman. And for secrets, like a password, you tell the other person not to write it in the notebook. That's the never-indexed type. ## 8. Key Takeaways - HTTP/1.1 is text. The server scans for `\r\n` and uses `Content-Length` to find where the body ends. HTTP/2 is binary: everything is a frame with a fixed 9-byte header (Length, Type, Flags, Reserved, Stream ID). - Type tells the server what the frame is (`0x01` HEADERS, `0x00` DATA). Stream ID tells it which request the frame belongs to. A HEADERS frame always comes first. - HPACK compresses only the HEADERS frame payload. The frame header and DATA frames are left alone. - HPACK uses a static table (61 fixed entries), a dynamic table (built per connection, starts at index 62, oldest entries dropped first) and optional Huffman coding (a fixed table, shorter codes for common characters). - The first bits of each entry say how to read it: `1` indexed, `01` literal and add to table, `0000` literal and don't add, `0001` literal and never add. - Client and server never exchange their dynamic tables. They stay in sync by reading the same header blocks in the same order. - In the RFC example, a repeated `:authority` header goes from 14 bytes to 1 byte. The saving builds up over many requests, and the first request on every new connection is the most expensive. - HTTP/2 still runs on a single TCP connection. If TCP loses a packet, every stream on that connection waits. So HTTP/2 fixes head-of-line blocking at the HTTP level, but not at the TCP level. ## 9. Next Topic Next up: the rest of what I promised in the last post, HTTP/2 head of line blocking and QUIC, which exist because of the TCP-level blocking HTTP/2 couldn't fix.