Every time a packet leaves your Linux machine, it travels through a surprisingly long sequence of stages. Understanding this path helps enormously when debugging network issues.
The socket layer
When your application calls send() or write() on a socket, the kernel’s socket layer takes over. For a TCP socket this means handing the data to tcp_sendmsg(), which in turn enqueues it into the socket’s send buffer.
You can observe the send queue depth with ss -tipm:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 2048 10.0.0.5:8080 192.168.1.20:47284
If Send-Q is consistently high, the application is sending faster than the network can drain.
TCP layer
The TCP layer segments data into MSS-sized chunks, adds the TCP header with sequence numbers, and hands segments to the IP layer. The kernel uses CUBIC by default on Linux; BBR is available and often better for high-bandwidth paths.
sysctl net.ipv4.tcp_congestion_control
Switch to BBR:
echo 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.d/99-net.conf
sudo sysctl -p /etc/sysctl.d/99-net.conf
IP and netfilter
Packets then enter the IP layer. Here netfilter hooks provide filtering (via iptables or nftables), NAT, and mangling. The path through hooks is: PREROUTING -> INPUT/FORWARD -> POSTROUTING.
Queueing discipline
After netfilter, the packet hits the qdisc layer. The default is fq_codel on modern kernels, which provides fair queuing with active queue management.
tc qdisc show dev eth0
Driver and NIC
The driver finally pushes the packet to the NIC hardware via the ring buffer. Modern NICs offload checksums, TCP segmentation (TSO), and receive steering (RSS).
ls /sys/class/net/eth0/queues/
ethtool -l eth0
Measuring end-to-end
The best tool for observing all these layers at once is bpftrace. A simple one-liner to count TCP retransmissions by process:
bpftrace -e 'kprobe:tcp_retransmit_skb { @[comm] = count(); }'
For serious performance work, perf combined with netstat, ss, and ethtool stats gives a complete picture.