sodcalculator

10 Tech How-To

How Linux Processes Packets Before Apps

Guide 10-49057 · Updated · Step by step

Linux receives network packets, validates and transforms them through a fixed kernel pipeline, and only then delivers the final data to an application via a socket. This path—from network interface card (NIC) through kernel subsystems to the user-space process—involves hardware interrupts, softirqs, protocol handlers, and socket buffers. Understanding this journey explains why packet processing is fast, why some packets are dropped, and how tools like tcpdump and firewalls hook into the flow.

The Big Picture: From Wire to Socket

When a packet arrives at a Linux machine, it does not magically appear in the application’s memory. The kernel acts as a middleman, performing several tasks in a strict order:

  1. Reception – The NIC copies the packet into RAM and signals the CPU.
  2. Interrupt handling – The kernel acknowledges the hardware and schedules processing.
  3. Protocol decoding – The packet is parsed (Ethernet, IP, TCP/UDP) and validated.
  4. Netfilter hooks – Firewall rules and other kernel modules inspect or modify the packet.
  5. Socket delivery – The data is queued to the receiving socket’s buffer.
  6. Wake-up – The application is notified that data is ready to read.

Each step is designed to minimize copying and CPU usage, but every layer adds a small cost. The entire process typically takes microseconds on modern hardware.

Step 1: Hardware Reception and Interrupts

The network interface card (NIC) listens on the wire (or Wi-Fi) and receives raw bits. It assembles them into a frame, checks the Ethernet CRC, and uses Direct Memory Access (DMA) to write the frame into a ring buffer in system RAM. The NIC then raises a hardware interrupt (IRQ) to tell the CPU that new data is available.

The kernel’s interrupt handler runs with interrupts disabled, so it must be quick. It typically does three things:

  • Acknowledges the interrupt to the NIC.
  • Marks the received packet as needing softirq processing.
  • Returns, allowing other interrupts to fire.

To avoid flooding the CPU, modern drivers use New API (NAPI), which disables further interrupts while packets are being processed in a poll loop. This reduces overhead under high traffic.

Step 2: Softirq and the Network Stack

After the hard interrupt, the kernel schedules a softirq—a deferred interrupt context that runs with interrupts enabled. The network softirq (NET_RX_SOFTIRQ) calls the driver’s poll function, which processes packets from the ring buffer.

For each packet, the kernel creates a struct sk_buff (socket buffer), which is the central data structure for network packets in Linux. It contains pointers to the data, metadata about the protocol, and timestamps. The sk_buff is passed through the protocol layers:

  • Ethernet layer – Checks the frame type (IPv4, IPv6, ARP) and strips the Ethernet header.
  • IP layer – Validates the IP header checksum, checks the destination address, and handles fragmentation if needed.
  • Transport layer – For TCP, checks the sequence numbers and updates the connection state; for UDP, simply validates the checksum.

This is also where netfilter hooks are invoked. Netfilter is the framework behind iptables, nftables, and firewalld. Hooks exist at multiple points: PREROUTING (before routing), LOCAL_IN (for packets destined to the host), FORWARD (for routed packets), and LOCAL_OUT (for locally generated packets). Each hook allows kernel modules to inspect, drop, modify, or redirect packets.

Step 3: Routing and Local Delivery

After the IP layer, the kernel consults the routing table to decide what to do with the packet. If the destination IP is one of the machine’s own addresses, the packet is marked for local delivery. If not, it is forwarded to another interface (if IP forwarding is enabled) or dropped.

For local delivery, the kernel looks up the socket that matches the packet’s protocol, source IP, destination IP, and port numbers. This is done via a hash table to be fast. If no matching socket exists, the kernel sends an ICMP port-unreachable message (for UDP) or sends a TCP RST (for TCP) and drops the packet.

If a matching socket is found, the packet’s payload is appended to the socket’s receive queue. For TCP, the kernel also handles flow control, congestion control, and reassembly of out-of-order segments before data is made available.

Step 4: Socket Buffer and Wake-up

The socket’s receive buffer is a bounded queue. If the application is not reading fast enough, the buffer fills up. When that happens, the kernel applies backpressure:

  • For TCP, it shrinks the advertised receive window, telling the sender to slow down.
  • For UDP, it simply drops new packets (UDP has no flow control).

Once data is queued, the kernel wakes up any process blocked in a read(), recv(), or select() call. The application then copies data from kernel memory to its own buffer via the copy_to_user operation. This copy is unavoidable in the standard path, though zero-copy mechanisms like splice() or AF_XDP can bypass it.

Common Problems and Misconceptions

One common misconception is that a packet is processed immediately when it arrives. In reality, the kernel batches packets to improve efficiency. NAPI polls multiple packets per interrupt, and the softirq runs on a per-CPU basis, which can lead to uneven load if traffic is heavy on one CPU.

Another issue is receive livelock, where the CPU spends all its time processing interrupts and never gets to user-space tasks. Modern kernels mitigate this with NAPI and by limiting the number of packets processed per poll cycle.

Packet drops are also frequently misunderstood. A drop can happen at many stages:

  • NIC ring buffer full (driver-level).
  • Socket buffer full (application not reading).
  • Firewall rule (netfilter).
  • IP checksum failure or invalid header.

Tools like ethtool -S and netstat -i show drop counters on the NIC and interface, while ss -s shows socket-level statistics.

How Tools Hook Into the Path

Packet capture tools like tcpdump and Wireshark use the AF_PACKET socket type, which receives a copy of every packet at the Ethernet layer. This happens after the NIC driver but before the IP stack processes the packet. That’s why tcpdump can see packets that are later dropped by iptables.

Firewalls like iptables and nftables hook into netfilter, as described earlier. They can drop packets before they reach a socket, so an application never sees them. Conversely, a packet can be accepted by the firewall but still be dropped because the socket buffer is full.

Performance tools like perf and bpftrace can trace kernel functions to see exactly where time is spent. For example, perf record -e skb:consume_skb shows when packets are freed.

Performance Optimizations and Alternatives

For high-throughput applications, the standard path may be too slow. Several optimizations exist:

  • RSS (Receive Side Scaling) – Distributes packets across multiple CPUs by hashing flows.
  • RPS/RFS – Software versions of RSS that can spread load on systems without multi-queue NICs.
  • Busy polling – The application spins on the NIC’s ring buffer instead of sleeping, reducing latency.
  • AF_XDP – A socket type that allows user-space to read packets directly from the ring buffer, bypassing the kernel protocol stack entirely.
  • DPDK – A user-space library that takes full control of the NIC, bypassing the kernel altogether.

These are advanced techniques used in routers, firewalls, and high-frequency trading. For most applications, the standard kernel path is sufficient.

Frequently Asked Questions

Why does tcpdump show packets that iptables drops?

tcpdump uses AF_PACKET sockets, which receive a copy of the packet at the Ethernet layer, before netfilter hooks are invoked. So the packet is captured even if iptables later drops it. This is useful for debugging, but it means tcpdump does not reflect what the application actually receives.

What is the difference between a hard interrupt and a softirq?

A hard interrupt is a hardware signal that pauses the CPU and runs a handler with interrupts disabled. A softirq is a deferred software interrupt that runs later with interrupts enabled, allowing other work to proceed. The network stack uses both to minimize the time spent with interrupts disabled.

How can I see if packets are being dropped?

Use ethtool -S eth0 to see NIC-level drop counters, netstat -i for interface-level errors and drops, and ss -s for socket-level statistics. Also check /proc/net/softnet_stat for drops in the softirq layer.

Does the kernel copy the packet data multiple times?

In the standard path, the NIC uses DMA to copy data from the wire into RAM (no CPU copy). The kernel then copies data from the socket buffer to user space when the application reads. That’s one CPU copy. Zero-copy techniques like splice() or AF_XDP avoid even that copy.

Why does UDP drop packets more easily than TCP?

UDP has no flow control or retransmission. If the socket buffer is full, new packets are simply dropped. TCP, on the other hand, advertises a receive window and slows down the sender, so data is not lost (unless the system is severely overloaded).

Can a packet be processed by multiple CPUs?

Yes, with RSS or RPS, different flows can be handled by different CPUs. However, a single flow (e.g., one TCP connection) is typically pinned to one CPU to avoid out-of-order processing. This is why you may see one CPU busy while others are idle under heavy traffic.

Related

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →