Diagnosing High Server Load Without Guessing

"The server is slow." It's the vaguest ticket you'll ever get, and the temptation is always the same: restart something and see if it helps. Sometimes it does, which is the worst outcome — because now you've destroyed the evidence and learned nothing.

Here's the triage order I use. It's deliberately mechanical, because the whole point is to stop guessing.

Step 0: understand what load average actually means

Load average is the number of processes that are either running or waiting to run — and on Linux, uniquely, it also counts processes blocked on uninterruptible I/O. That last part is why load can hit 60 on a machine whose CPUs are almost idle.

uptime
# 14:22:01 up 91 days,  load average: 42.15, 38.02, 21.77

nproc
# 8

Compare load to core count. Load 8 on 8 cores is fully utilised but coping. Load 42 on 8 cores means a deep queue. And the three numbers are 1, 5 and 15-minute averages — if the first is much higher than the third, it's getting worse right now; the reverse means you're looking at the tail of something that has already passed.

Step 1: CPU, I/O, or memory?

One command answers this. Run top and read the CPU line — not the process list, the summary line.

%Cpu(s):  4.2 us,  1.1 sy,  0.0 ni, 12.3 id, 81.9 wa,  0.0 hi,  0.5 si

In the example above, wa is 81.9%. That's not a CPU problem — that's a disk problem, and no amount of PHP tuning will help.

Step 2a: it's CPU

# Top CPU consumers, cleanly
ps -eo pid,ppid,user,pcpu,pmem,etime,cmd --sort=-pcpu | head -20

# Per-core breakdown — one pegged core is a single-threaded bottleneck
mpstat -P ALL 2 5

# Which user, on a shared server
ps -eo user,pcpu --no-headers | awk '{c[$1]+=$2} END {for (u in c) print c[u], u}' | sort -rn | head

On a hosting server, the usual suspects in order of likelihood:

• One site under a bot crawl. Not malicious, just an aggressive crawler hitting uncached dynamic pages. Check access logs by request rate per vhost.
• A PHP process stuck in a loop. Look for a process with hours of CPU time. strace -p PID tells you what it's actually doing.
• An XML-RPC or login brute force. Cheap for the attacker, expensive for you — every attempt is a full PHP bootstrap.
• A cryptominer. Sustained 100% CPU from an oddly-named process running as a web user. This is a compromise, not a performance ticket — switch tracks.

# Which vhost is getting hammered
tail -10000 /var/log/apache2/access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head

# Requests per minute, to see if it's a burst or sustained
awk '{print substr($4,2,17)}' /var/log/apache2/access_log | uniq -c | tail -20

Step 2b: it's I/O

High wa means processes are queued on storage. Find out which device and which process.

# Per-device utilisation. %util near 100 = saturated device
iostat -xz 2 5

# Which processes are doing the I/O
iotop -oPa

# Processes currently in uninterruptible sleep (blocked on I/O)
ps -eo state,pid,user,cmd | awk '$1 ~ /D/'

In iostat, the columns that matter:

• %util — how busy the device is. Sustained near 100% means saturated.
• await — average time a request waits, in ms. On SSD, anything above ~10ms is suspicious; on spinning disk, above ~50ms.
• aqu-sz — average queue depth. A consistently deep queue means requests are backing up faster than the device drains them.

Common causes on a hosting box, most frequent first:

• An unindexed database query doing full table scans on a large table. This is the number one cause I see, by a wide margin.
• A backup job running in business hours. Check cron before you go looking for anything cleverer.
• Swapping. If memory is exhausted, the disk activity is a symptom of the memory problem — jump to the next section.
• A full or nearly-full disk. Filesystems slow down badly above ~90% as allocation gets harder.
• A failing disk. Check dmesg for I/O errors and SMART for reallocated sectors before assuming it's a workload problem.

df -h                       # full disk?
df -i                       # out of inodes? (many small files)
dmesg -T | grep -iE 'i/o error|ata|nvme'
smartctl -H /dev/sda

Step 2c: it's memory

Memory pressure is the sneakiest of the three, because it disguises itself as an I/O problem: the machine starts swapping, disk saturates, and wa goes through the roof.

free -h

# si/so columns: if these are non-zero and sustained, you're swapping
vmstat 2 5

# Biggest RSS consumers
ps -eo pid,user,rss,cmd --sort=-rss | head -15

# Has the OOM killer been active?
dmesg -T | grep -i 'killed process'

On a web server, memory exhaustion is nearly always one of:

• PHP-FPM pm.max_children set too high for the RAM available. Each child can hold tens of megabytes; multiply it out and check the total actually fits.
• MySQL buffer pool oversized relative to the box — a config copied from a machine with more RAM.
• A genuine application leak — one process whose RSS grows steadily over days.

# Average RSS per php-fpm child, in MB
ps -ylC php-fpm --sort:rss | awk '{s+=$8; n++} END {print s/n/1024 " MB avg over " n-1 " procs"}'

# Multiply that by pm.max_children and compare to free memory.
# If it exceeds available RAM, you are one traffic spike from the OOM killer.

Step 3: fix the cause, not the symptom

Now — and only now — act. Match the fix to what you actually found:

Step 4: make the next one easier

The incident isn't over when load drops. Two things stop it recurring:

• An alert on the actual leading indicator you just discovered — I/O wait percentage, available memory, slow query count. Not just load average, which told you nothing about the cause.
• A note in the runbook. "High wa on web-07 is usually the reporting query on account X" is the kind of institutional knowledge that turns a two-hour investigation into a two-minute one.

The whole method fits on an index card: read the CPU summary line, decide whether it's CPU, I/O or memory, then find the specific process. Three commands, in order, before you change anything. It's slower than restarting Apache, and it's the only approach that stops the ticket coming back.