Posts on Besterry — Linux & DevOps Notes

Incident Response Playbook That Actually Gets Used

Sun, 22 Dec 2024 00:00:00 +0000

Most incident playbooks end up as wiki pages nobody reads during an actual incident. Here’s what survives contact with a real 3am pager.

The first five minutes

One person is the Incident Commander (IC). If that’s not clear, declare yourself IC.

Acknowledge the page
Post in #incidents: “Incident: [brief]. I am IC.”
Start a timeline document (even just a text file)
Check public status page — update if user-visible

Don’t dig into the problem yet. Set up the command structure first.

The Observability Pyramid: Logs, Metrics, Traces in 2026

Tue, 10 Dec 2024 00:00:00 +0000

The three pillars of observability are talked about a lot. Which one to reach for depends on the question you’re answering.

Metrics: for “is it broken and how much”

Aggregated numerical data over time. Good for:

Dashboards and alerts
Trends (is latency increasing week-over-week?)
Capacity planning

Not good for:

Explaining why a specific request was slow
Finding causality between events

Stack: Prometheus + Grafana remains the default. OpenTelemetry Metrics if you want vendor-neutral instrumentation.

Rust vs Go for CLI Tools: A Practical Comparison

Mon, 25 Nov 2024 00:00:00 +0000

After writing CLI tools in both Rust and Go over the last few years, here are the things that actually matter when choosing between them.

Startup time

Go wins. A trivial Go program starts in ~1-5ms. A trivial Rust program also starts in ~1-5ms. Both are negligible for CLI tools. (The old argument about Go’s startup was mostly about JVM-vs-Go, not Go-vs-Rust.)

Binary size

Out of the box:

Go: 5-15 MB for a small program
Rust: 2-8 MB for a small program (with LTO and strip)

After aggressive optimization:

tcpdump Filters Cheatsheet for When the Network is On Fire

Fri, 08 Nov 2024 00:00:00 +0000

tcpdump has a weird little filter language (BPF syntax) that I never remember under pressure. This page is my cheatsheet.

Basic syntax

tcpdump -i <interface> -n <filter>
-n don't resolve addresses/ports
-i interface (eth0, any, lo)
-v verbose (-vv, -vvv more)
-w write to file for later wireshark
-r read from file
-c N stop after N packets
-s 0 capture full packet (not truncated)

Host and network filters

host 192.0.2.1 # to or from
src host 192.0.2.1 # from only
dst host 192.0.2.1 # to only
net 192.0.2.0/24 # subnet
src net 192.0.2.0/24 # subnet as source

Port filters

port 443 # source or dest port 443
src port 443 # source only
dst port 443 # dest only
portrange 50000-60000 # range

Protocol filters

tcp # TCP only
udp # UDP only
icmp # ICMP only
arp # ARP
tcp port 443 # combine
'tcp[tcpflags] & tcp-syn != 0' # TCP with SYN flag

TCP flag combinations

# SYN only (connection attempts)
'tcp[tcpflags] == tcp-syn'
# SYN-ACK
'tcp[tcpflags] == tcp-syn|tcp-ack'
# RST (connection resets)
'tcp[tcpflags] & tcp-rst != 0'
# FIN (connection closes)
'tcp[tcpflags] & tcp-fin != 0'

Combining filters

host 192.0.2.1 and tcp port 443
'host 192.0.2.1 and (port 80 or port 443)'
'not arp and not port 22'

Boolean operators: and, or, not (or &&, ||, !).

Self-Host vs SaaS: The Actual Tradeoffs

Tue, 22 Oct 2024 00:00:00 +0000

The “self-host everything” movement has passionate advocates on both sides. Reality is nuanced. Here’s the framework I use when deciding.

Cost isn’t the main factor

Many self-host advocates lead with cost savings. Usually it’s misleading:

SaaS at small scale is often free or cheap ($0-50/mo)
Self-hosting on cheap VPS starts around $5/mo
But self-hosting eats engineer time — 2-10 hours/month for maintenance
At $100/hr engineering time, self-hosting often costs MORE than SaaS

Cost-wise, self-hosting wins when you’re either:

Grafana Dashboards That Don't Suck: Principles and Anti-Patterns

Sat, 05 Oct 2024 00:00:00 +0000

Most Grafana dashboards are bad. Too many panels, unclear queries, inconsistent color schemes, no clear purpose. Here are the principles I apply now.

Rule 1: Every dashboard has one question

Start by writing down: “What question does this dashboard answer?”

Good:

“Is the order service healthy right now?”
“How is the nightly ETL job progressing?”
“What is the cost trend for our compute in the last 30 days?”

Bad:

“Production metrics”
“Database overview”

If you can’t state the question in one sentence, you don’t know what the dashboard is for.

Terraform State Locking: Why You Need It and How It Goes Wrong

Wed, 18 Sep 2024 00:00:00 +0000

Terraform state without locking is a bug waiting to happen. Two engineers running apply simultaneously can corrupt state in ways that take hours to untangle. Here’s what I learned after one such incident.

Why state locking matters

Terraform reads state, computes a plan, and writes new state. Without locking, two concurrent runs can:

Both read the same initial state
Both compute their plans based on it
Both write conflicting state — last one wins
Now state doesn’t match real infrastructure

The symptoms are weird: resources exist but Terraform wants to create them again. Or state references resources that were already destroyed.

ZFS on Linux: Six Months of Production Use

Mon, 02 Sep 2024 00:00:00 +0000

Migrated our build server array from ext4+mdadm to ZFS on Linux six months ago. Here’s what I learned.

Why ZFS

Checksumming catches silent data corruption (we found 14 affected files on the old array)
Snapshots are cheap and instant (100ms for a 10TB dataset)
Compression often makes things faster — less I/O, more CPU
Send/receive for efficient replication
No separate mdadm/LVM layer to debug

Pool design

For the build server, 6 x 4TB NVMe in RAIDZ2:

PostgreSQL Backup Strategies: Not All Backups Are Equal

Sun, 18 Aug 2024 00:00:00 +0000

A backup you can’t restore isn’t a backup. After losing data once (fortunately from a test environment), here’s the framework I apply now.

The three levels of recovery

Point-in-time recovery (PITR): Restore to any second in the last N days. Requires WAL archiving + base backups.
Daily snapshots: Restore to yesterday’s 3am state. Simple, cheap, 24h RPO.
Logical dumps: Restore specific tables or data subsets. Useful for selective recovery.

Most production databases should have all three.

Modern TLS Cipher Configuration in 2026

Mon, 05 Aug 2024 00:00:00 +0000

Configuring TLS ciphers used to involve copying a magic list from Mozilla SSL Configurator and moving on. In 2026 the landscape has shifted enough that revisiting is worth it.

What changed

TLS 1.3 is now supported by 95%+ of clients. Serving TLS 1.0 or 1.1 is an active liability.
OpenSSL 3.x became the default on most modern distros. Some older ciphers are simply gone.
Post-quantum hybrid key exchange (X25519-Kyber768) started rolling out in Chrome and Firefox.
Perfect Forward Secrecy is universally expected. No more RSA key exchange.

Recommended nginx config

ssl_protocols TLSv1.2 TLSv1.3;
# TLS 1.3 cipher suites (nginx picks automatically)
ssl_ciphers 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
ssl_prefer_server_ciphers off;
ssl_ecdh_curve X25519:secp521r1:secp384r1;

ssl_prefer_server_ciphers off is correct for modern deployments — clients know better than servers which ciphers perform well on their hardware.

Kubernetes Troubleshooting: The First 10 Minutes of an Outage

Mon, 22 Jul 2024 00:00:00 +0000

When PagerDuty wakes you up about a Kubernetes cluster issue, the first 10 minutes matter. Here is the runbook I work through before anything else.

Get your bearings

First, confirm what’s actually broken from the user side. Check the status page or synthetic monitor. Many “outages” are monitoring issues, not real problems.

Cluster-level check

kubectl get nodes
kubectl top nodes

Look for NotReady nodes and resource pressure. If multiple nodes are down, the problem is probably infrastructure — check the cloud provider console.

Alert Fatigue: Prometheus Rules That Actually Help

Mon, 10 Jun 2024 00:00:00 +0000

Most alerts are noise. The hardest part of monitoring is deciding what NOT to alert on. Here is the framework I use.

Rule 1: Every alert must be actionable

If you get paged and there is nothing to do, the alert should not exist. Either fix the root cause, automate the response, or let it be a metric trend instead of a page.

Rule 2: Alert on user-visible symptoms

Instead of HighCPUUsage, prefer HighRequestLatency. CPU usage high with good latency means the system is working as designed. Latency high means users are hurting.

Reducing Container Image Size: Multi-Stage Builds and Alpine

Mon, 20 May 2024 00:00:00 +0000

Small images boot faster, save bandwidth, and have smaller attack surface. Here are the techniques that actually work.

Multi-stage builds

The single biggest win. Build in one stage, copy only the artifacts to a minimal runtime stage. A Go binary of 15 MB ends up in a 17 MB image. Compare to a naive golang:1.22 image at 900+ MB.

Base image choice

From smallest to largest for Go/Rust static binaries:

Useful bpftrace One-Liners for System Debugging

Thu, 02 May 2024 00:00:00 +0000

bpftrace makes the kernel event space accessible from a bash one-liner. Here are the scripts I keep reaching for.

Count syscalls by process

bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

Distribution of file read sizes

bpftrace -e 'tracepoint:syscalls:sys_enter_read { @ = hist(args->count); }'

TCP retransmissions by remote address

bpftrace -e '
kprobe:tcp_retransmit_skb {
$sk = (struct sock *)arg0;
$daddr = $sk->__sk_common.skc_daddr;
@[ntop($daddr)] = count();
}'

Process creation stream

bpftrace -e 'tracepoint:sched:sched_process_exec { printf("%s\n", str(args->filename)); }'

When to use bpftrace vs perf vs strace

strace: simple, but adds significant overhead. Fine for debugging a single misbehaving process.
perf: best for sampling-based profiling (CPU time, cache misses). Low overhead.
bpftrace: best for event-driven tracing across the whole system. Tiny overhead if used sparingly.

All three should be in your toolbox.

WireGuard vs AmneziaWG: When Obfuscation Matters

Mon, 15 Apr 2024 00:00:00 +0000

Plain WireGuard is simple and fast. AmneziaWG adds obfuscation to the handshake. When do you need which?

Plain WireGuard is enough when

You control both endpoints, no DPI is filtering your traffic, and the main concern is performance and simplicity. WireGuard shines for:

Site-to-site VPN between your own servers
Remote access to a home lab
Point-to-point tunnels on a LAN

The handshake is small, fast, and provably secure. It uses Noise framework primitives and 1 RTT.

SSH Hardening Checklist for Public VPS

Mon, 01 Apr 2024 00:00:00 +0000

Every public-facing server gets port-scanned within minutes of going online. Default SSH settings are decent but not great. Here is the checklist I run through on every new VPS.

Disable password authentication

In /etc/ssh/sshd_config:

PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no

PermitRootLogin prohibit-password

This allows root login with key but not password, which is fine for automation. For stricter setups, use no and sudo from an unprivileged user.

Docker Network Debugging: nsenter and tcpdump Patterns

Wed, 20 Mar 2024 00:00:00 +0000

When a container cannot reach something, the instinct is often to exec into it and curl. But most slim containers lack curl, dig, tcpdump, or even ping. A better pattern: use nsenter from the host.

Enter the container network namespace

Get the container PID:

docker inspect -f '{{.State.Pid}}' myapp

Then:

sudo nsenter -t PID -n bash

You are now in the container network namespace, but with the host binaries. tcpdump, ip, ss, dig — all work.

nginx Performance Tuning: Practical Notes from Production

Tue, 05 Mar 2024 00:00:00 +0000

After running nginx on everything from 512 MB VPS instances to multi-socket bare metal, here are the settings I’ve found actually matter.

worker_processes and worker_connections

Start with worker_processes auto;.

worker_processes auto;
worker_rlimit_nofile 65535;
events {
worker_connections 4096;
use epoll;
multi_accept on;
}

Keepalive tuning

http {
keepalive_timeout 30s;
keepalive_requests 1000;
upstream backend {
server 10.0.0.1:8080;
keepalive 32;
}
}

Buffer sizes

client_body_buffer_size 128k;
client_max_body_size 50m;
proxy_buffer_size 8k;
proxy_buffers 8 8k;

gzip and brotli

gzip on;
gzip_comp_level 5;
gzip_types text/plain text/css application/json;
brotli on;
brotli_comp_level 4;
brotli_types text/plain text/css application/json;

Measurement

None of this matters if you don’t measure. Install nginx-module-vts or expose stub_status, feed metrics to Prometheus, and compare before/after for any changes.

systemd Timers vs Cron: When to Use Which

Sat, 17 Feb 2024 00:00:00 +0000

Cron has been the standard scheduler on Unix for decades. systemd timers are newer, more powerful, but also more verbose.

Cron wins when

Cron is perfect for one-line scripts that need to run on a simple schedule. Writing:

0 3 * * * /usr/local/bin/backup.sh

is fast, requires no other files, and works on every Unix-like system since the 1970s.

systemd timers win when

You want any of these:

Logging integrated with journalctl
Dependencies on other units (After=network-online.target)
Resource limits (MemoryMax=, CPUQuota=)
Randomized delays to avoid thundering herd (RandomizedDelaySec=)
The ability to manually trigger with systemctl start
Catch-up behavior after system was off (Persistent=true)

Minimal systemd timer example

/etc/systemd/system/backup.service:

Linux Networking Deep Dive: From Socket to Wire

Sat, 10 Feb 2024 00:00:00 +0000

Every time a packet leaves your Linux machine, it travels through a surprisingly long sequence of stages. Understanding this path helps enormously when debugging network issues.

The socket layer

When your application calls send() or write() on a socket, the kernel’s socket layer takes over. For a TCP socket this means handing the data to tcp_sendmsg(), which in turn enqueues it into the socket’s send buffer.

You can observe the send queue depth with ss -tipm: