Why does an injected Node.js agent from an eBPF tracer stay inside a process after the service is excluded, until a pod restart?

Short answer: because the injected JS agent is a one-time, process-lifetime attachment that OBI's exclusion rules do not undo. OBI discovers Node.js processes and injects a JS agent into each of them through the Node inspector protocol (SIGUSR1 + Runtime.evaluate); the injection is a queued action that runs long after discovery accepted the process, and once the agent is in, it lives for the rest of that process's life. exclude_instrument and friends are discovery-time selection rules — they gate which processes get instrumented, not whether the already-installed agent gets removed. So excluding a service after testing stops OBI from instrumenting new process incarnations, but the pods that are already running still carry the agent (with its per-callback event-loop cost) until those pods restart and a fresh process incarnation is created. That is why a rollout restart is what actually makes the latency drop.

The two lifetimes that don't match

The confusion comes from two different objects with different lifetimes:

1. The OBI config's exclusion rules (discovery.exclude_instrument, and per-rule exclude:). These are selection criteria evaluated at discovery time — the same definition format as the inclusion selectors. They answer "which processes should OBI pick up and instrument?", and the docs position them for exactly this use: "avoid instrumenting services typically found in observability environments," or excluding a port-forwarder by executable path. Nothing in these rules is a command to processes.
2. The injected agent inside the process. The injector's design comment states the timing directly: "Injection is queued and runs long after that" discovery accepted the process, and the target is pinned by PID + start-time ("a stable reference to the process incarnation") so that "the numeric PID alone would let a recycled one be identified, signaled and injected in the original's place." Injection is a one-shot event against a process incarnation; it is not a continuously maintained subscription.

Once the agent is in, its lifetime is the process's lifetime. The fdextractor.js agent even restores the original net prototypes at the top of every re-injection run (so re-injection doesn't double-wrap), and its runtime-metrics cleanup "stays outside the gate: a re-injection with runtime metrics disabled must tear down what a previous injection installed" — but nothing in the shipped path removes the agent in response to a config change. The agent does not know OBI excluded the service; it just keeps running the hooks it was installed to run.

Why exclusion is a gate, not a retractor

exclude_instrument follows the same definition format as the inclusion selectors (namespace, labels, executable path, open ports…), and OBI's docs point to it for dropping services it should never instrument — observability stack components, a Docker/socat forwarder that shares a port with the real service. Semantically that is a discovery filter: when OBI (re)scans for processes to instrument, an excluded service is simply not picked. For a service you excluded after it had been instrumented:

• OBI stops picking up new incarnations of it;
• already-running pods keep the agent, with its async_hooks before hook and fs.existsSync sentinel still firing on the event loop;
• a restart (or the pod's natural process lifecycle) creates a new incarnation that discovery now excludes — and only then is the per-callback cost gone.

So the rollout restart is not a workaround; it is the mechanism by which a discovery-time exclusion becomes an effect on running processes. The eBPF side behaves the same way: the uprobes attach to process incarnations, and an excluded service is no longer attached to when it next starts — the running one is what you have to recycle.

How to verify which half you are looking at

1. Check the config is actually in place before you test anything. Confirm the exclude_instrument / exclude selectors (or OTEL_EBPF_TARGET_PID overrides) match the service, then verify on a newly started pod that OBI does not instrument it — that is the half the config controls.
2. Check the running pods separately. A pod that was instrumented before the exclusion still carries the agent. The decisive signal is the process's age relative to the config change: anything started before it still pays the per-callback cost, anything started after it does not. A latency comparison that mixes the two will read as "the exclusion did not work."
3. Profile the event loop of an excluded-but-already-injected pod. If p99 still shows the per-callback sentinel cost, the agent is still resident; the fix is to recycle the pod (or, going forward, the agent-uninstall path currently in in-flight work — a closed draft PR that switches the sentinel to fs.existsSync and adds "uninstall the agent on shutdown," which gives the agent an explicit way off a running process rather than relying on process death).
4. Do not treat "still expensive after exclusion" as a discovery bug. It is the expected consequence of exclusion being a discovery-time gate. The unresolved boundary is that the shipped flow has no live retraction of an already-installed agent; that is exactly what the in-flight uninstall work is for.

The limitation that decides it

exclude_instrument (and per-rule exclude:) decide which discovery events OBI acts on; they do not send a "remove yourself" message to processes that already carry the injected agent. Injection is a one-time, incarnation-scoped event — pinned to a PID + start-time because the PID alone could be recycled — so the agent's removal follows the process's death, not the config's wording. The decisive boundary: exclusion takes effect at the next process incarnation, not in place. The practical consequence is that excluding a service mid-testing still requires a rollout restart of the affected pods to realize the latency improvement; the follow-through is to keep exclusion rules in place for the new incarnations and, where the agent-uninstall work lands, use it to stop paying the cost without a restart.

References

• OBI docs: Service discovery (exclude_instrument — "Specify selection criteria for excluding services from being instrumented", same definition format as the inclusion selectors; exclude: per rule)
• OBI docs: Troubleshooting (excluding a port-forwarder from instrumentation by executable path — the exclusion gate in practice)
• OBI source: injection_target.go (injection "is queued and runs long after" discovery; target pinned by PID + start-time because "the numeric PID alone would let a recycled one be identified, signaled and injected in the original's place")
• OBI source: fdextractor.js (agent restores the original net prototypes on every re-injection; runtime-metrics "Cleanup stays outside the gate: a re-injection with runtime metrics disabled must tear down what a previous injection installed")
• OBI draft PR #3357: nodejs sentinel performance + "uninstall the agent on shutdown" (closed, not merged — the explicit way off a running process that does not exist in the shipped flow)
• Node.js docs: async_hooks (the before hook the injected agent installs; marked experimental, AsyncLocalStorage recommended on performance grounds)

Community discussion today

The monitored window is a rolling week across two allowlisted Slack archives (both technical); two visible-browser chat workspaces and public mailing-list/forum surfaces were not reviewed this run, and that gap is noted rather than treated as quiet. Several themes recurred from the prior days' windows; the technically new angle is the agent lifecycle.

Node.js agent lifecycle and exclusion (the question above). The Node sentinel cost-attribution thread continued with a maintainer-confirmed read: client-span parenting uses the fd-pair map while the per-callback sentinel keeps the trace-context map aligned with the active request, and an additional consumer is external trace/profile correlation — so gating the sentinel only on manual spans + log enrichment could silently break that integration. One user then reported that after excluding a service from tracing they had to roll out a pod restart of all pods to see latency drop, inferring that the tracing process was "left hanging inside pods even after the exclusion." That report is the lifecycle boundary this page answers: the agent is a one-time, incarnation-scoped attachment, so a discovery-time exclusion cannot retract it from a running process — the restart is what removes it. A maintainer's direction (separate per-callback refresh from fd-pair correlation, gate per consumer) and the closed, unmerged draft PR for sentinel performance + uninstall-on-shutdown mean there is no shipped live-retraction path yet; the unresolved boundary is whether OBI will gain an explicit agent-removal trigger.

OBI config v1→v2 migration and the Helm chart gap (recurring). A user still hitting "fields are outside the supported v1-to-v2 migration contract" on their config; the maintainer's position is unchanged: the all-or-nothing migrate is intentional (a silently-dropped field can produce a valid-looking but materially different config), and the proposed middle ground is an explicit --allow-partial / --best-effort mode that migrates what it can and reports the omitted fields — not a silent default. The related OBI Helm-chart v2 gap (a v1 field still injected into the v2 config schema by the chart's _helpers.tpl) was directed to be filed as a separate issue in the helm-charts repo, including chart/OBI versions, the relevant values, the rendered config, and the validation error.

Managed agent harnesses and observability lock-in (recurring). The thread on the hosted Agents API and the general "vendor holds the telemetry" pattern: when the agent loop runs in a managed harness, the inner span tree only exists if the harness exports it, and a public beta that exposes no tracing configuration or external trace exporters means generic HTTP-client instrumentation can only reconstruct the outbound crossings. A practitioner proposed metering model/tool calls as an ordinary HTTP receipt header so existing HTTP-client spans capture it without inventing new attribute vocabulary; a GenAI agent span-shape thread (one agent span per turn with model calls and tool executions as siblings, correlated by the tool-call identifier) was answered against the current agent-span conventions.