eBPF from first principles

The first time I really understood eBPF was when I stopped thinking of it as “a way to do fast networking” and started thinking of it as what it literally is: a small register virtual machine baked into the Linux kernel, plus a static analyzer that refuses to load any program it can’t prove is safe. Everything else — Cilium replacing kube-proxy, bpftrace one-liners, Falco watching syscalls — is built on those two facts. You can write code, hand it to the kernel, and the kernel will run it in kernel space at native speed after first proving it can’t loop forever, read memory it shouldn’t, or panic the machine.

That last clause is the whole trick. A kernel module can do anything; a bad one corrupts memory and takes the box down with it. An eBPF program is the opposite bargain: you give up generality, and in exchange the kernel guarantees the thing can’t hurt it. This post is my notes on how that bargain actually works underneath — the VM, the verifier, the maps, the helpers, and the spectrum of places you can hook.

The virtual machine

eBPF is a register-based VM. It has 11 64-bit registers (R0–R10), a program counter, and a 512-byte stack. The registers aren’t arbitrary scratch space — they have a calling convention burned into the design:

• R0 holds return values from helper calls and the program’s own exit value.
• R1–R5 pass arguments to helper functions. On program entry, R1 holds a pointer to the context (an skb for a packet program, a pt_regs for a kprobe, and so on).
• R6–R9 are callee-saved: their values survive across helper calls.
• R10 is a read-only frame pointer to the top of the stack. You can read it and compute offsets from it, but you can’t write to it.

This is a deliberate echo of real 64-bit hardware (roughly the x86-64 / arm64 calling conventions), which is exactly why the JIT step later can map BPF registers almost one-to-one onto machine registers. The instruction set is small and fixed-width: load/store, ALU ops, jumps, and a call instruction. There’s no printf, no arbitrary memory, no syscalls from inside. The VM is intentionally anemic.

This is the “e” in eBPF — extended BPF. The original, now called cBPF (classic BPF), was the Berkeley Packet Filter: two 32-bit registers and a tiny instruction set whose entire job was deciding whether tcpdump should keep a packet. Extended BPF kept the spiritual idea (a safe little filter VM in the kernel) and blew it up into a general-purpose-ish execution engine: 64-bit registers, maps, helper calls, and dozens of attach points that have nothing to do with packets. Internally the kernel still calls all of it “BPF”; cBPF programs are now transparently translated to eBPF before they run.

The lifecycle: source to running code

A program goes through a fixed pipeline before a single instruction executes on a hook. You write restricted C, clang/LLVM (which has a bpf backend target) compiles it to BPF bytecode in an ELF object file, you load it through the bpf() syscall, the verifier scrutinizes it, the JIT turns it into native code, and finally you attach it to a hook where the kernel will run it on every relevant event.

The crucial property is that the verifier runs at load time, once, not on every execution. By the time the program is attached, it’s already been proven safe and compiled to native instructions, so the runtime cost on the hot path is just running native code. You pay the safety tax up front.

The verifier is the whole game

If you only understand one component, make it this one. The verifier is what makes eBPF fundamentally different from a kernel module, and almost every frustrating “why won’t my program load” moment traces back to it.

It works in two broad phases. First, a control-flow check. The verifier builds the program’s control-flow graph and does a DAG check: it rejects unreachable instructions and, historically, rejected any back-edge — i.e. loops — outright. The program had to be a directed acyclic graph that always marched toward an exit.

Second, symbolic execution. Starting from the first instruction, the verifier walks every reachable path through the program and simulates it, tracking the state of every register and every stack slot as it goes. It doesn’t run the program; it tracks what it knows about each value. Every register has a type — NOT_INIT (never written, therefore unreadable), SCALAR_VALUE (a number, not usable as a pointer), or one of several pointer types like PTR_TO_CTX, PTR_TO_MAP_VALUE, PTR_TO_STACK, PTR_TO_PACKET. The rules are strict and a little surprising the first time:

• A register that was never written to is not readable. R0 = R2 fails immediately if R2 was never initialized.
• Reading from the stack is only allowed after you’ve written to that slot.
• After a helper call, R1–R5 are scratched to unreadable and R0 takes the helper’s return type. R6–R9 survive.
• Pointer arithmetic is heavily constrained. Adding two pointers produces a SCALAR_VALUE (a now-useless number), because the result isn’t a meaningful address.

For scalars and pointer offsets, the verifier tracks ranges, not just types. For each value it keeps signed and unsigned min/max bounds and a “tnum” — a known-bits representation (a mask of unknown bits plus the known values). When you read a byte into a register, it knows the top 56 bits are zero and the low 8 are unknown. When you compare a value > 8 and take the true branch, it narrows the minimum to 9. This range tracking is how it proves memory accesses are in bounds.

Two more pieces make this tractable and bearable. State pruning: the verifier caches the register/stack state at instructions it has already analyzed. When it reaches that instruction again on another path, if a previously-accepted state is a superset of the current one (at least as general, at least as strict on alignment), it prunes the branch — the earlier acceptance implies this one is fine too. Liveness tracking of which registers actually get used later makes more states equivalent and prunes harder. Without this, the path explosion would be exponential.

And the complexity limit: the verifier will only analyze up to 1 million instructions total across all paths before giving up. This is why a logically-fine program can still be rejected for being “too complex” — you didn’t write an infinite loop, you just gave the verifier more branches than it’s willing to explore. The practical program size limit for unprivileged-style loads has historically been 4,096 instructions; privileged programs can be far larger, bounded by that 1M analysis budget.

Bounded loops arrived in kernel 5.3. Before that, every loop had to be manually unrolled (#pragma unroll) so the program stayed a DAG. Since 5.3 the verifier can accept a real loop as long as it can prove the loop has an exit condition that’s guaranteed to become true — it simulates iterations and prunes states until the loop’s induction variable provably terminates. You still can’t write an unbounded while (1).

The thing to internalize: the verifier is a safety tool, not a security tool. It proves the program can’t crash or read out of bounds. It does not reason about whether what the program does is a good idea.

Helpers: you can’t just call kernel functions

A BPF program can’t call arbitrary kernel functions. If it could, every program would be welded to one exact kernel version’s internal symbols, and a typo could jump anywhere. Instead the kernel exposes a stable, curated set of helper functions — things like bpf_map_lookup_elem(), bpf_ktime_get_ns(), bpf_get_current_pid_tgid(), bpf_probe_read_kernel(), bpf_perf_event_output(). The call instruction dispatches to these by number, the verifier checks that R1–R5 match the helper’s declared argument constraints, and R0 comes back with a known return type.

Which helpers a program is allowed to call depends on its program type. A socket filter and a tracing probe see different helper sets, because what’s safe to expose differs by context. This whitelisting is a load-time decision baked into the verifier’s per-type configuration.

The newer mechanism is kfuncs — kernel functions explicitly annotated as callable from BPF. Unlike the fixed helper ABI (which is treated as a stable contract), kfuncs are not promised to be stable across versions; they let the kernel expose functionality faster without committing to a frozen API forever. Combined with BTF (below), the verifier can type-check kfunc calls properly. Modern BPF increasingly leans on kfuncs rather than minting new numbered helpers.

Maps: the only way to hold or share state

A BPF program’s stack is 512 bytes and vanishes when the program returns. To keep state across invocations, share data between two BPF programs, or talk to userspace, you use maps — typed key/value stores that live in kernel memory and outlive any single program run. Both BPF programs (via helpers) and userspace (via the bpf() syscall) can read and write them. Maps are the communication channel.

The map type picks the data structure and the semantics:

• Hash and array are the workhorses — arbitrary keys, or integer-indexed slots.
• Per-CPU hash/array keep a separate copy of each value per CPU. The point is to avoid locking: a program running on CPU 3 only ever touches CPU 3’s copy, so concurrent updates from other cores can’t race. Userspace sums the per-CPU values when it reads. This is how you build lock-free counters at packet rates.
• LRU hash evicts least-recently-used entries when full, so a bounded map can track an unbounded key space (flow tracking, for instance) without growing forever.
• LPM trie does longest-prefix matching — the natural structure for routing tables and CIDR lookups.
• Ring buffer (and the older per-CPU perf buffer) stream variable-sized events up to userspace. The ring buffer, added in 5.8, is a single MPSC buffer shared across CPUs with proper ordering, which fixed the perf buffer’s per-CPU memory waste and event-reordering quirks.
• Map-of-maps and program array maps hold references to other maps or to programs (the latter powers tail calls, below).

Attach points: a spectrum from the wire to the syscall

A program does nothing until it’s attached to a hook, and the program type you compile determines both where it can attach and what its context (R1 on entry) looks like. The interesting mental model is a spectrum: the earlier in the stack you hook, the faster and cheaper you run, but the less context you have.

Going roughly bottom to top:

• XDP (eXpress Data Path) runs in the NIC driver the moment a packet arrives, before the kernel allocates an sk_buff. That’s why it’s the fastest place to drop, redirect, or rewrite packets — there’s no per-packet socket-buffer overhead yet. It’s the basis of DDoS scrubbing and L4 load balancing. The tradeoff is you only have the raw packet bytes and almost no kernel context.
• tc / clsact hooks attach at traffic-control ingress and egress, after the sk_buff exists. Slightly later than XDP, but you get the full packet metadata and you can hook egress too (XDP is ingress-only on most drivers). This is where Cilium does a lot of its pod-to-pod policy work.
• Socket-layer hooks (socket filters, sockops, cgroup/connect hooks) operate at the socket boundary — e.g. rewriting a destination at connect() time before any packet is even formed.
• kprobes / kretprobes attach to (almost) any kernel function entry or return by patching the instruction stream. Maximum flexibility, but they’re tied to internal function names and signatures, so they’re fragile across kernel versions.
• Tracepoints are stable, named instrumentation points the kernel maintainers commit to keeping. Less flexible than kprobes, far more durable.
• fentry / fexit are the modern replacement for kprobes on function entry/exit. Built on BTF and trampolines, they’re faster than kprobes and give you typed access to function arguments — but they require BTF (below).
• LSM hooks let a BPF program make security decisions at Linux Security Module checkpoints (this is what Tetragon and KRSI-style enforcement use).
• uprobes / perf events reach up into userspace function calls and hardware performance counters respectively.

CO-RE: compile once, run everywhere

Here’s the portability problem. A tracing program that reads task->pid needs the byte offset of pid within struct task_struct. That offset differs between kernel versions and configs. The old bcc approach shipped LLVM and kernel headers onto every target box and recompiled the program at runtime against the local headers. It worked, but dragging a compiler and headers onto every production node is heavy and slow.

CO-RE (Compile Once – Run Everywhere) fixes this with two pieces. BTF (BPF Type Format) is compact type information describing kernel structs and their layout; modern kernels ship their own BTF at /sys/kernel/btf/vmlinux. When you compile with CO-RE, clang emits relocations: instead of hard-coding “offset 0x4e8”, the object records “the offset of field pid in struct task_struct.” At load time, the loader library (libbpf) reads the running kernel’s BTF, resolves each relocation to the correct offset for this kernel, and patches the bytecode before handing it to the verifier. One compiled binary adapts itself to whatever kernel it lands on, no on-target compiler required. This is the single biggest reason eBPF tooling became practical to ship as ordinary binaries.

Tail calls and the size ceiling

Because a single program is capped (4,096 instructions in the classic limit, and the 1M verifier budget overall), you sometimes need to chain logic. Tail calls let one program jump into another via bpf_tail_call(), indexing into a special program-array map. It’s a jump, not a call — execution transfers entirely and does not return, much like execve() replacing a process image. This lets you build state machines and dispatch tables (parse the packet, then tail-call into the per-protocol handler) without blowing any one program’s size budget. Tail-call chains are themselves bounded — the kernel caps the chain depth (33 in current kernels) so you can’t recurse forever.

A program you can actually read

Here’s a minimal XDP program that counts received packets into a per-CPU array map and lets every packet through. It’s close to the canonical “hello world” of the data path:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} pkt_count SEC(".maps");

SEC("xdp")
int count_packets(struct xdp_md *ctx) {
    __u32 key = 0;
    __u64 *count = bpf_map_lookup_elem(&pkt_count, &key);
    if (count)            // verifier forces this NULL check
        __sync_fetch_and_add(count, 1);
    return XDP_PASS;      // hand the packet to the stack
}

char _license[] SEC("license") = "GPL";

The if (count) is not defensive style — it’s mandatory. bpf_map_lookup_elem returns PTR_TO_MAP_VALUE_OR_NULL, and the verifier will reject any dereference until you’ve branched on NULL, at which point the pointer’s type narrows to PTR_TO_MAP_VALUE and the access becomes provably safe. Omit the check and the load fails with R0 invalid mem access 'map_value_or_null'.

For pure observability you rarely write C at all. bpftrace compiles an awk-like one-liner straight to bytecode. This counts every execve by process name:

bpftrace -e 'tracepoint:syscalls:sys_enter_execve { @[comm] = count(); }'

@[comm] is a map keyed by process name; count() is the aggregation. Under the hood it’s a tracepoint program writing a hash map that userspace drains and prints on exit. Same machinery as the XDP example, completely different ergonomics.

Where this actually shows up

The big production users all sit on the same foundation. Cilium implements Kubernetes networking, service load-balancing, and identity-based policy in tc and XDP programs, storing service-to-backend maps in BPF hash maps — it can replace kube-proxy entirely. Katran (Meta’s L4 load balancer) is XDP doing consistent-hash backend selection at the NIC. bpftrace and bcc are the observability toolkits built on kprobes, tracepoints, and uprobes. Falco and Tetragon watch syscalls and LSM hooks for runtime security, turning kernel events into security signals. Different domains, one substrate.

The safety bargain, restated

The reason all of this is allowed to run in kernel space comes back to the verifier. A kernel module that dereferences a bad pointer panics the machine. A BPF program cannot get that far, because the verifier proved at load time that the pointer was bounds-checked, the loop terminates, the map lookup was NULL-checked, and no register was read before it was written. After verification, the JIT compiles it to native code and the kernel marks that memory read-only and hardens it against Spectre-style leaks. You get kernel-speed custom code with a static guarantee that a bug in it degrades to “the program is rejected” rather than “the box is down.”

---

What stuck with me, after poking at this in a homelab and reading more verifier source than I’d like to admit, is that eBPF isn’t really one technology — it’s a small VM, an unusually thorough static analyzer, a set of typed shared-memory regions, and a curated syscall-like surface, all wearing one name. The VM is almost boring; the analyzer is the entire reason any of it is safe to ship. Once you see the verifier as the load-bearing wall, the rest of the design stops looking arbitrary: maps exist because the stack is tiny and transient, helpers exist because you can’t call into the kernel freely, CO-RE exists because struct layouts move, and the attach-point spectrum exists because “where you hook” is just a trade between speed and context. It’s the same lesson as cgroups and namespaces — the magic is a stack of small, well-scoped primitives, and the moment you can name each one, “eBPF” stops being a buzzword and starts being a thing you can reason about.