eBPF Tutorial by Example: Learning CO-RE eBPF Step by Step

This is a development tutorial for eBPF based on CO-RE (Compile Once, Run Everywhere). It provides practical eBPF development practices from beginner to advanced, including basic concepts, code examples, and real-world applications. Unlike BCC, we use frameworks like libbpf, Cilium, libbpf-rs, and eunomia-bpf for development, with examples in languages such as C, Go, and Rust.

This tutorial does not cover complex concepts and scenario introductions. Its main purpose is to provide examples of eBPF tools (very short, starting with twenty lines of code!) to help eBPF application developers quickly grasp eBPF development methods and techniques. The tutorial content can be found in the directory, with each directory being an independent eBPF tool example.

For the complete source code of the tutorial, please refer to the repo https://github.com/eunomia-bpf/bpf-developer-tutorial on GitHub. If you find this tutorial helpful, please give us a star!

Getting Started Examples

This section contains simple eBPF program examples and introductions. It primarily utilizes the eunomia-bpf framework to simplify development and introduces the basic usage and development process of eBPF.

lesson 0-introduce Introduction to Core Concepts and Tools
lesson 1-helloworld Hello World, Framework and Development
lesson 2-kprobe-unlink Monitoring unlink System Calls with kprobe
lesson 3-fentry-unlink Monitoring unlink System Calls with fentry
lesson 4-opensnoop Capturing Opening Files and Filter with Global Variables
lesson 5-uprobe-bashreadline Capturing readline Function Calls with Uprobe
lesson 6-sigsnoop Capturing Signal Sending and Store State with Hash Maps
lesson 7-execsnoop Capturing Process Execution, Output with perf event array
lesson 8-exitsnoop Monitoring Process Exit Events, Output with Ring Buffer
lesson 9-runqlat Capturing Scheduling Latency and Recording as Histogram
lesson 10-hardirqs Capturing Interrupts with hardirqs or softirqs

Advanced Documents and Examples

We start to build complete eBPF projects mainly based on libbpf and combine them with various application scenarios for practical use.

lesson 11-bootstrap Develop User-Space Programs with libbpf and Trace exec() and exit()
lesson 12-profile Using eBPF Program Profile for Performance Analysis
lesson 13-tcpconnlat Statistics of TCP Connection Delay with libbpf
lesson 14-tcpstates Recording TCP Connection Status and TCP RTT
lesson 15-javagc Capturing User-Space Java GC Duration Using USDT
lesson 16-memleak Monitoring Memory Leaks
lesson 17-biopattern Count Random/Sequential Disk I/O
lesson 18-further-reading More Reference Materials： papers, projects
lesson 19-lsm-connect Security Detection and Defense using LSM
lesson 20-tc tc Traffic Control
lesson 21-xdp Programmable Packet Processing with XDP

In-Depth Topics

This section covers advanced topics related to eBPF, including using eBPF programs on Android, possible attacks and defenses using eBPF programs, and complex tracing. Combining the user-mode and kernel-mode aspects of eBPF can bring great power (as well as security risks).

GPU:

lesson 47-cuda-events Tracing CUDA GPU Operations
xpu flamegraph Building a GPU Flamegraph Profiler with CUPTI
lesson xpu/npu-kernel-driver Tracing Intel NPU Kernel Driver Operations
lesson xpu/gpu-kernel-driver Monitoring GPU Driver Activity with Kernel Tracepoints

Scheduler:

lesson 44-scx-simple Introduction to the BPF Scheduler
lesson 45-scx-nest Implementing the scx_nest Scheduler

Networking:

lesson 23-http L7 Tracing with eBPF: HTTP and Beyond via Socket Filters and Syscall Tracepoints
lesson 29-sockops Accelerating Network Request Forwarding with Sockops
lesson 41-xdp-tcpdump Capturing TCP Information with XDP
lesson 42-xdp-loadbalancer XDP Load Balancer
lesson 46-xdp-test Building a High-Performance XDP Packet Generator

Tracing:

lesson 30-sslsniff Capturing SSL/TLS Plain Text Data Using uprobe
lesson 31-goroutine Using eBPF to Trace Go Routine States
lesson 33-funclatency Measuring Function Latency with eBPF
lesson 37-uprobe-rust Tracing User Space Rust Applications with Uprobe
lesson 39-nginx Using eBPF to Trace Nginx Requests
lesson 40-mysql Using eBPF to Trace MySQL Queries
lesson 48-energy Energy Monitoring for Process-Level Power Analysis

Security:

lesson 24-hide Hiding Process or File Information
lesson 25-signal Using bpf_send_signal to Terminate Malicious Processes in eBPF
lesson 26-sudo Privilege Escalation via File Content Manipulation
lesson 27-replace Transparent Text Replacement in File Reads
lesson 28-detach Running eBPF After Application Exits: The Lifecycle of eBPF Programs
lesson 34-syscall Modifying System Call Arguments with eBPF

Features:

lesson 35-user-ringbuf Asynchronously Send to Kernel with User Ring Buffer
lesson 36-userspace-ebpf Userspace eBPF Runtimes: Overview and Applications
lesson 38-btf-uprobe Expanding eBPF Compile Once, Run Everywhere(CO-RE) to Userspace Compatibility
lesson 43-kfuncs Extending eBPF Beyond Its Limits: Custom kfuncs in Kernel Modules
features bpf_wq BPF Workqueues for Asynchronous Sleepable Tasks
features struct_ops Extending Kernel Subsystems with BPF struct_ops
features dynptr BPF Dynamic Pointers for Variable-Length Data
features bpf_arena BPF Arena for Zero-Copy Shared Memory
features bpf_iters BPF Iterators for Kernel Data Export

Other:

lesson 49-hid Fixing Broken HID Devices Without Kernel Patches
cgroup cgroup-based Policy Control

Android:

lesson 22-android Using eBPF Programs on Android

Continuously updating...

eBPF Tutorial by Example 0: Introduction to Core Concepts and Tools

This is the first part of a comprehensive development tutorial for eBPF, designed to guide you through practical eBPF development, from beginner to advanced. It covers fundamental concepts, real-world code examples, and

eBPF Tutorial by Example 1: Hello World, Framework and Development

In this blog post, we will delve into the basic framework and development process of eBPF (Extended Berkeley Packet Filter). eBPF is a powerful network and performance analysis tool that runs on the Linux kernel, providi

eBPF Tutorial by Example 2: Monitoring unlink System Calls with kprobe

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code at runtime.

eBPF Tutorial by Example 3: Monitoring unlink System Calls with fentry

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and execute user-defined code at runtime in the kernel.

eBPF Tutorial by Example 4: Capturing Opening Files and Filter with Global Variables

eBPF (Extended Berkeley Packet Filter) is a kernel execution environment that allows users to run secure and efficient programs in the kernel. It is commonly used for network filtering, performance analysis, security mon

eBPF Tutorial by Example 5: Capturing readline Function Calls with Uprobe

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel that allows developers to dynamically load, update, and run user-defined code at runtime.

eBPF Tutorial by Example 6: Capturing Signal Sending and Store State with Hash Maps

eBPF Tutorial by Example 7: Capturing Process Execution, Output with perf event array

eBPF Tutorial by Example 8: Monitoring Process Exit Events, Output with Ring Buffer

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code at runtime in the kernel.

eBPF Tutorial by Example 9: Capturing Scheduling Latency and Recording as Histogram

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool on the Linux kernel. It allows developers to dynamically load, update, and run user-defined code at runtime.

eBPF Tutorial by Example 10: Capturing Interrupts with hardirqs or softirqs

eBPF Tutorial by Example 11: Develop User-Space Programs with libbpf and Trace exec() and exit()

eBPF Tutorial by Example 12: Using eBPF Program Profile for Performance Analysis

This tutorial will guide you on using eBPF programs for performance analysis with a Rust implementation. We will leverage the perf mechanism in the kernel to learn how to capture the execution time of functions and view

eBPF Tutorial by Example 13: Statistics of TCP Connection Delay with libbpf

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting the ke

eBPF Tutorial by Example 14: Recording TCP Connection Status and TCP RTT

eBPF (Extended Berkeley Packet Filter) is a powerful network and performance analysis tool widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without restarting

eBPF Tutorial by Example 15: Capturing User-Space Java GC Duration Using USDT

eBPF Tutorial by Example 16: Monitoring Memory Leaks

eBPF (extended Berkeley Packet Filter) is a powerful network and performance analysis tool that is widely used in the Linux kernel. eBPF allows developers to dynamically load, update, and run user-defined code without re

eBPF Tutorial by Example 17: Count Random/Sequential Disk I/O

eBPF (Extended Berkeley Packet Filter) is a new technology in the Linux kernel that allows users to execute custom programmes in kernel space without changing the kernel code. This provides system administrators and deve

The Secure Path Forward for eBPF runtime: Challenges and Innovations

Extended Berkeley Packet Filter (eBPF) represents a significant evolution in the way we interact with and extend the capabilities of modern operating systems. As a powerful technology that enables the Linux kernel to run

More Reference Materials： papers, projects

You may find more about eBPF in these places:

eBPF Tutorial by Example 19: Security Detection and Defense using LSM

eBPF Tutorial by Example 20: tc Traffic Control

Linux's Traffic Control (tc) subsystem has been present in the kernel for many years. Similar to the relationship between iptables and netfilter, tc includes a user-space tc program and a kernel-level traffic control fra

eBPF Tutorial by Example 21: Programmable Packet Processing with XDP

In this tutorial, we will introduce XDP (eXpress Data Path) and walk through a small example to help you get started. Later on, we will explore more advanced XDP applications, such as load balancers, firewalls, and other

eBPF Tutorial by Example: Using eBPF Programs on Android

This article mainly documents the author's exploration process, results, and issues encountered while testing the level of support for CO-RE technology based on the libbpf library on high version Android kernels in the A

L7 Tracing with eBPF: HTTP and Beyond via Socket Filters and Syscall Tracepoints

In today's technology landscape, with the rise of microservices, cloud-native applications, and complex distributed systems, observability of systems has become a crucial factor in ensuring their health, performance, and

eBPF Practical Tutorial: Hiding Process or File Information

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows you to run, load, and update user-defined code without having to change the kernel source code or reboot the kernel. This capab

Using bpfsendsignal to Terminate Malicious Processes in eBPF

eBPF (Extended Berkeley Packet Filter) is a revolutionary technology in the Linux kernel that allows users to execute custom programs in kernel space without modifying the kernel source code or loading any kernel modules

eBPF Tutorial: Privilege Escalation via File Content Manipulation

eBPF's power extends far beyond simple tracing—it can modify data flowing through the kernel in real-time. While this capability enables innovative solutions for performance optimization and security monitoring, it also

eBPF Tutorial: Transparent Text Replacement in File Reads

When you read a file in Linux, you trust that what you see matches what's stored on disk. But what if the kernel itself was lying to you? This tutorial demonstrates how eBPF programs can intercept file read operations an

Running eBPF After Application Exits: The Lifecycle of eBPF Programs

eBPF Development Practices: Accelerating Network Request Forwarding with Sockops

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows running, loading, and updating user-defined code without the need to modify the kernel source code or reboot the kernel. This c

eBPF Practical Tutorial: Capturing SSL/TLS Plain Text Data Using uprobe

With the widespread use of TLS in modern network environments, tracing microservices RPC messages has become increasingly challenging. Traditional traffic sniffing techniques often face limitations in accessing only encr

eBPF Practical Tutorial: Using eBPF to Trace Go Routine States

Go, the popular programming language created by Google, is known for its powerful concurrency model. One of the key features that makes Go stand out is the use of goroutines—lightweight, managed threads that make it easy

eBPF Tutorial by Example 32: Wall Clock Profiling with Combined On-CPU and Off-CPU Analysis

Performance bottlenecks can hide in two very different places. Your code might be burning CPU cycles in hot loops, or it might be sitting idle waiting for I/O, network responses, or lock contention. Traditional profilers

Measuring Function Latency with eBPF

In modern software systems, understanding the performance characteristics of functions—especially those critical to the operation of your application—is paramount. One key metric in performance analysis is function laten

eBPF Development Practice: Modifying System Call Arguments with eBPF

eBPF (Extended Berkeley Packet Filter) is a powerful feature in the Linux kernel that allows user-defined code to be run, loaded, and updated without the need to modify kernel source code or reboot the kernel. This funct

eBPF Development Practices: Asynchronously Send to Kernel with User Ring Buffer

eBPF, or Extended Berkeley Packet Filter, is a revolutionary technology in the Linux kernel that allows developers to run custom "micro programs" in kernel mode, thereby changing system behavior or collecting fine-graine

Userspace eBPF Runtimes: Overview and Applications

In this blog post, we'll dive into the world of eBPF in userspace. While many are familiar with kernel-based eBPF, userspace eBPF runtimes have been making significant strides and offer compelling use cases. We will also

eBPF Practice: Tracing User Space Rust Applications with Uprobe

eBPF, or Extended Berkeley Packet Filter, is a revolutionary technology in the Linux kernel that allows developers to run custom "micro-programs" in kernel mode, thus changing system behavior or collecting granular perfo

Expanding eBPF Compile Once, Run Everywhere(CO-RE) to Userspace Compatibility

eBPF, short for extended Berkeley Packet Filter, is a powerful and versatile technology used in modern Linux systems. It allows for the running of sandboxed programs in a virtual machine-like environment within the kerne

using BTF to verify userspace eBPF extensions

Here we will show how to use the type information of userspace application to verify the eBPF program which will access the userspace memory(include valid or invalid data structure memory access, pointer access, etc), an

Using eBPF to Trace Nginx Requests

Nginx is one of the most popular web servers and reverse proxies in the world, known for its high performance, stability, and low resource consumption. It is widely used for serving static content, load balancing, and ac

Using eBPF to Trace MySQL Queries

MySQL is one of the most widely used relational database management systems in the world. Whether you are running a small application or a large-scale enterprise system, understanding the performance characteristics of y

eBPF Tutorial by Example: Capturing TCP Information with XDP

Extended Berkeley Packet Filter (eBPF) is a revolutionary technology in the Linux kernel that allows developers to run sandboxed programs within the kernel space. It enables powerful networking, security, and tracing cap

Network setup for bpf-developer-tutorial

In this tutorial, we will set up a simple network topology that simulates a load balancer using eBPF/XDP (Express Data Path). The setup includes a local machine, a load balancer (which can be enhanced with an XDP program

eBPF Developer Tutorial: XDP Load Balancer

In this tutorial, we will guide you through the process of implementing a simple XDP (eXpress Data Path) load balancer using eBPF (Extended Berkeley Packet Filter). With just C, libbpf, and no external dependencies, this

Extending eBPF Beyond Its Limits: Custom kfuncs in Kernel Modules

Have you ever felt constrained by eBPF's capabilities? Maybe you've run into situations where the existing eBPF features just aren't enough to accomplish your goals. Perhaps you need deeper interactions with the kernel,

eBPF Tutorial: Introduction to the BPF Scheduler

Welcome to our deep dive into the world of eBPF with a focus on the BPF scheduler! If you're looking to extend your eBPF knowledge beyond the basics, you're in the right place.

eBPF Tutorial by Example: Implementing the scxnest Scheduler

In the ever-evolving landscape of system performance optimization, the ability to customize and extend kernel behavior is invaluable. One of the most powerful tools for achieving this is eBPF (extended Berkeley Packet Fi

eBPF Tutorial by Example: Building a High-Performance XDP Packet Generator

Need to stress-test your network stack or measure XDP program performance? Traditional packet generators like pktgen require kernel modules or run in userspace with high overhead. There's a better way - XDP's BPFPROGRUN

eBPF Tutorial: Tracing CUDA GPU Operations

Have you ever wondered what's happening under the hood when your CUDA application is running? GPU operations can be challenging to debug and profile because they happen in a separate device with its own memory space. In

eBPF Tutorial: Energy Monitoring for Process-Level Power Analysis

Have you ever wondered how much energy your applications are consuming? As energy efficiency becomes increasingly critical in both data centers and edge devices, understanding power consumption at the process level is es

eBPF Tutorial: Fixing Broken HID Devices Without Kernel Patches

Ever plugged in a new mouse or drawing tablet only to find it doesn't work quite right on Linux? Maybe the Y-axis is inverted, buttons are mapped wrong, or the device just feels broken. Traditionally, fixing these quirks

The bpftrace One-Liner Tutorial

This teaches you bpftrace for Linux in 12 easy lessons, where each lesson is a one-liner you can try running. This series of one-liners introduces concepts which are summarized as bullet points. For a full reference to b

eBPF Tutorial: cgroup-based Policy Control

Do you need to enforce network access control on containers or specific process groups without affecting the entire system? Or do you need to restrict certain processes from accessing specific devices while allowing othe

eBPF Tutorial by Example: BPF Arena for Zero-Copy Shared Memory

Ever tried building a linked list in eBPF and got stuck using awkward integer indices instead of real pointers? Or needed to share large amounts of data between your kernel BPF program and userspace without expensive sys

eBPF Tutorial: BPF Iterators for Kernel Data Export

Ever tried monitoring hundreds of processes and ended up parsing thousands of /proc files just to find the few you care about? Or needed custom formatted kernel data but didn't want to modify the kernel itself? Tradition

eBPF Tutorial: BPF Workqueues for Asynchronous Sleepable Tasks

Ever needed your eBPF program to sleep, allocate memory, or wait for device I/O? Traditional eBPF programs run in restricted contexts where blocking operations crash the system. But what if your HID device needs timing d

eBPF Tutorial by Example: BPF Dynamic Pointers for Variable-Length Data

Ever written an eBPF packet parser and struggled with those verbose dataend bounds checks that the verifier still rejects? Or tried to send variable-length events through ring buffers only to find yourself locked into fi

eBPF Tutorial: Extending Kernel Subsystems with BPF structops

Have you ever wanted to extend kernel behavior—like adding a custom scheduler, network protocol, or security policy—but were put off by the complexity of writing and maintaining a full kernel module? What if you could de