Why does every HTTP/2 request after the first on a TLS connection fail HPACK decoding when captured with TLS uprobes, and how should a decoder keep per-connection state?

The failure is not a decoding bug. HTTP/2 field compression (HPACK) is stateful, and the state is a property of the connection, not of the individual TLS record or event that a capture pipeline emits. The RFC 9113 field-block rule says that field compression is "stateful. Each endpoint has an HPACK encoder context and an HPACK decoder context ... that is used for all field blocks on a connection", and HTTP/2 says that "a decoding error in a field block MUST be treated as a connection error" — specifically COMPRESSION_ERROR when the decoder "is unable to maintain the field section compression context" (RFC 9113, §4.3 / §7.2). RFC 7541 makes the ownership concrete: the dynamic table is "specific to an encoding or decoding context", the static and dynamic tables start out empty on a fresh context, and indexed header field names can address the combined static-plus-dynamic table. A decoder that is rebuilt for every uprobe event (or for every worker that picks up an event) has an empty dynamic table, so the first header block on a connection that happens to use only static or literal fields decodes, and the first header block that indexes into the dynamic table the peer has built since connection setup is a decoding error. One decoder instance per connection, fed the events of that connection in byte order, is the requirement — and any lost event has to mark that connection's decoder unreliable, not be silently replayed or skipped.

The mechanism

The HPACK state machine that a capture-side decoder has to mirror, from RFC 7541:

• §2.2: the encoder's and decoder's dynamic tables are "completely independent" — the encoder (your peer's TLS stack) builds its dynamic table from what it actually sends, and the decoder has to rebuild the same table by consuming the header blocks in order.
• §2.3.2: the decoder's dynamic table is specific to its decoding context, initially empty, and maintained as a FIFO: "as each header block is decompressed", indexed entries get added to the head and, when the table size is exceeded, the least-recently-used entries are evicted from the tail.
• §2.3.3 and §4.1: the index space is static-table entries followed by dynamic-table entries; each dynamic entry consumes 32 bytes of entry size plus name and value, within the dynamic table size (the HTTP/2 SETTINGS_HEADER_TABLE_SIZE default is 4096).
• §3.2: any malformed header block is a decoding error, and HTTP/2 escalates that to a connection error. §6.1 adds that an index of 0 is unused and an index that exceeds the static-plus-dynamic table size is an error.

So the failure mode is structural. When the capture pipeline constructs a fresh decoder for each TLS record written by gnutls_record_send (or for each event delivered to a worker), the decoder's dynamic table is empty at the start of every record. Request #1 on the connection typically references only static-table fields (method, path, host) and decodes. Request #2's header block references a dynamic-table index the peer's encoder created in an earlier header block — which the fresh decoder never saw — and the decoder reports an invalid indexed representation. Every subsequent request on that connection fails the same way, and the peer's own error is the HTTP/2 COMPRESSION_ERROR connection error, which is exactly the public symptom seen in eCapture's long-running issue 744, where HTTP/2 tracing over TLS lost every request packet after the first on a TCP flow. The diagnosis there is the same mechanism: "the deep reason is hpack dynamic table cannot be shared" between event handlers, and the attempted fixes (keying per-connection decoders on the 4-tuple or on the struct sock pointer, one event worker per connection with a shared dynamic table) are the same design space below.

That issue also documents the keying trap: a struct sock pointer is not a stable identity, because the kernel socket allocator (sk_prot_alloc in net/core/sock.c) has no documented guarantee that a freed address is not reused for a new socket. The same reasoning applies, with the same lack of documented guarantee, to the GnuTLS session pointer: gnutls_session_t is the first argument of gnutls_record_send, so it is trivially available in the uprobe event, but there is no documented guarantee against session-pointer reuse after the session is freed. GnuTLS's own teardown path, gnutls_deinit, is documented to "clear all buffers and remove session data from the session database" — that is where a per-connection decoder has to be dropped, not at some arbitrary LRU eviction. The robust identity is the transport: gnutls_transport_set_ptr(session, ptr) and gnutls_transport_set_int(session, fd) are the transport-binding entry points, and probing the latter plus gnutls_deinit gives you a mapping from (pid, transport fd) to decoder state that survives the session pointer going out of scope. One caveat worth knowing when you probe around GnuTLS: a TLS client such as curl that selects the GnuTLS backend binds its own internal context struct via gnutls_transport_set_ptr(session, cf) rather than an int fd, so on those processes the record functions carry no usable file descriptor in their arguments — the mapping probe has to key on the transport binding that actually happens, and that is observable through the same two transport functions.

Ordering is the second structural problem, and it is orthogonal to correctness: even with the right key, the decoder is only as good as the order and completeness of the header blocks it receives. A per-CPU perf event array read by several workers delivers events out of order, and an overflow drops records permanently. The kernel ring-buffer documentation motivates the ring buffer exactly for this: "more efficient memory utilization by sharing ring buffer across CPUs" plus "preserving ordering of events ... even across multiple CPUs", which perf buffers "fail to satisfy both" (the ring buffer is a multi-producer single-consumer ring). The capture-side design that matches HPACK's requirements is therefore:

1. One decoder state per connection, owned by the workers for that connection.
2. A single reader per connection (one worker drains that connection's events in byte order), not a pool that may split one connection's events across threads.
3. On any dropped, truncated, or out-of-order event, mark that connection's decoder unreliable and drop its subsequent header-block data rather than guessing. A decoder that "decodes without error" after a gap is not trustworthy — the gap has permanently changed the table the peer is indexing into, and there is no way to replay a dropped TLS record.

Verification and debugging path

The first thing to confirm is that the error is structural, not incidental. Rebuild the pipeline to keep one decoder per connection key and watch the failures vanish: if request #2 onwards decodes with per-connection state and still fails with per-event state, the dynamic-table mechanism is confirmed. Concretely:

1. Key the decoder state on the transport binding, not on the session pointer. Probe gnutls_transport_set_int (and gnutls_transport_set_ptr for ptr-bound stacks) and gnutls_deinit, and maintain (pid, transport fd) → decoder. Drop the decoder on deinit. This is the same shape as eCapture issue 744's per-connection worker proposal, with the identity fixed to something the kernel hands you.
2. Replace the multi-reader per-CPU perf array with one reader per connection — either a dedicated worker per connection, or a single consumer per CPU with per-connection queues in front. The kernel ring-buffer documentation states the property the design needs: ordering preserved across CPUs on a single consumer.
3. Add a liveness signal per connection: on a ring-buffer overrun or a sequence gap, set that connection's decoder to unreliable and stop emitting decoded headers for it until it reconnects (which resets the HPACK context). This is honest: a decoder that decodes without error after a gap is not trustworthy.
4. Verify against the known public symptom: a GnuTLS-backed client doing HTTP/2 over TLS should lose exactly the requests after the first per connection when the decoder is per-event, and decode all of them when the decoder is per-connection.

If you want to skip HPACK entirely, the alternative is to capture headers before the encoder: eBPF uprobes on the HTTP/2 library's request-submission API read the plaintext header array before compression. For a Go gRPC stack, Pixie's eBPF approach probes loopyWriter.writeHeader and reads the []hpack.HeaderField slice — the documented caveat is that the approach is specific to a single HTTP/2 library's internals and breaks across Go calling-convention changes. For libnghttp2-based stacks, the public analog is nghttp2_submit_request / nghttp2_submit_request2, whose signature takes an nghttp2_nv header array (session, pri_spec, nva, nvlen, data_prd, stream_user_data): the uprobe reads the array before the library HPACK-compresses it, and the same class of boundaries applies — the capture is of submitted headers, not of what the kernel actually transmitted, the symbol set varies with distribution packages and static linking, and each supported HTTP/2 library is a separate maintenance surface.

The Go reference decoder for cross-checking your implementation is golang.org/x/net/http2/hpack: NewDecoder takes a reader, the dynamic table defaults to 4096 bytes, and an index beyond the static-plus-dynamic table size surfaces as InvalidIndexError — the userspace-visible shape of the same decoding error the kernel-side or capture-side decoder hits.

The limitation

The hard limit is event loss, and it is unrecoverable. A dropped TLS record means the decoder has permanently missed header blocks that extended the peer's dynamic table; any subsequent "clean" decode is a coincidence, not correctness. The design consequence is asymmetric: correctness is a per-connection invariant you can engineer for (one state per connection, ordered feed, reliable drop signaling), but completeness across a lossy capture path is not, and the honest behavior is to mark the connection unreliable and stop emitting decoded data rather than to ship partial or guessed headers. That asymmetry is also the reason the eCapture text-mode implementation was closed rather than fixed: "The text mode has many flaws ... temporarily closed" — the flaws are exactly the ones the keying and ordering sections above are about.

References

• RFC 7541 — HPACK: Hyperframe Compression — §2.2 (independent encoder/decoder dynamic tables), §2.3.2 (per-context dynamic table, FIFO maintenance), §2.3.3 (index space), §3.2 (decoding errors), §4.1 (entry size), §6.1 (indexed representation).
• RFC 9113 — HTTP/2 — §4.3 / §4.3.1 (stateful field compression per connection; decoding errors are connection errors), §6.5.2 (SETTINGS_HEADER_TABLE_SIZE), §7.2 (COMPRESSION_ERROR).
• GnuTLS Manual — §6.5 transport functions (gnutls_transport_set_ptr / gnutls_transport_set_int), §6.7 record and teardown functions (gnutls_record_send, gnutls_bye, gnutls_deinit).
• eCapture issue 744 — HTTP/2 requests lost after the first per connection — public prior art: COMPRESSION_ERROR symptom, dynamic-table sharing diagnosis, 4-tuple/sock keying attempts, per-connection worker proposal.
• Pixie — eBPF-based HTTP/2 tracing — plaintext header capture via loopyWriter.writeHeader uprobes; library-specific maintenance boundary.
• nghttp2 API reference — nghttp2_submit_request / nghttp2_submit_request2 and the nghttp2_nv header array.
• golang.org/x/net/http2/hpack — NewDecoder, InvalidIndexError, default dynamic table size.
• Linux kernel — BPF ring buffer — MPSC ring, cross-CPU ordering and shared-memory motivation.

Community discussion today

The day's substantive eBPF signal came from two threads in the opt-in archives.

The first, and the one selected above, is an HTTPS sensor built on GnuTLS uprobes: gnutls_record_send capture plus a userspace perf reader and an HTTP/2 frame parser, where the first request on a TLS connection decodes and every later request fails COMPRESSION_ERROR — the per-connection HPACK decoder-state problem described in this answer. Its public prior art is eCapture issue 744, which carries the same symptom and the same diagnosis (the hpack dynamic table is not shareable across event handlers, RFC 7541 §2.2), plus the keying attempts and the closed-text-mode caveat.

The second is a LoadBalancer behavior thread: with a shared VIP, a second service claiming an already-owned frontend is rejected with "frontend already owned by another service" (the public error is ErrFrontendConflict in pkg/loadbalancer/errors.go, wrapped in pkg/loadbalancer/writer/writer.go), and the community's stated understanding is that deleting the owning service does not re-queue the claimant, while an annotation-only update on the claimant recovers the state in roughly seven seconds with no BPF flush — the suspected mechanism is event coalescing on the Kubernetes informer side. The public files the thread names are pkg/loadbalancer/writer/writer.go, pkg/loadbalancer/reflectors/k8s.go, and pkg/container/insert_ordered_map.go. Presented as the community's stated understanding, not as a verified diagnosis: the coalescing hypothesis has not been pinned to a code path in the public sources.

A third thread — a ring-0 socket-drop versus context-switch benchmark comparison — was too thin to select, and is noted here only as low-signal.

Channel coverage for this run: the visible-browser-only sources (Discord, the eunomia-bpf and sched-ext communities, the bpf mailing list, r/eBPF) could not be reviewed in this run — no visible-browser session was available — so they are marked uncovered, not quiet. Only the two opt-in archive sources were available and covered above.