Reading the Signs: Detecting a Compromised Server Early

By the time a customer calls to say their site has been defaced, the attacker has usually been on the server for weeks. Defacement is loud, and loud is the last thing an attacker does — because it ends the access they've been quietly monetising.

The useful signals come much earlier, and they're mostly boring: a load average that's slightly higher than it used to be, a cron job nobody wrote, mail queuing for domains you don't host. This is what to look for, and — just as important — the order to act in once you've found it.

The early signals

1. Outbound mail you didn't send

The most common monetisation of a compromised hosting account is spam. It shows up in the mail queue long before anyone notices anything else.

# Exim: queue size and top senders
exim -bpc
exim -bp | exiqsumm | head -20

# Postfix
mailq | tail -1
qshape deferred | head

# Which script sent it — the smoking gun on a shared server
grep 'cwd=' /var/log/exim_mainlog | awk -F'cwd=' '{print $2}' \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

That last command is the one I reach for first on cPanel servers. Exim logs the working directory of the script that called sendmail — so it points straight at the compromised account, and often the exact directory.

2. Processes that shouldn't exist

# Processes running from world-writable or temp directories
ls -l /proc/*/cwd 2>/dev/null | grep -E '/tmp|/dev/shm|/var/tmp'

# Deleted binaries still running — a classic hiding technique
ls -l /proc/*/exe 2>/dev/null | grep deleted

# Long-running CPU burners (miners look exactly like this)
ps -eo pid,user,etime,time,pcpu,cmd --sort=-time | head -15

# Network connections with owning process
ss -tupn state established

A process whose binary has been deleted but is still running is very rarely legitimate. Neither is a web user with an outbound connection to an unfamiliar IP on a high port.

3. Files that changed when nobody deployed

# PHP files changed in the last 48 hours under web roots
find /home/*/public_html -name '*.php' -mtime -2 -ls

# Files whose inode changed (ctime) — catches timestamp tampering,
# because an attacker who sets mtime back usually forgets ctime
find /home/*/public_html -name '*.php' -ctime -7 -ls

# Classic obfuscation markers
grep -rlE 'eval\s*\(|base64_decode\s*\(|gzinflate\s*\(|str_rot13\s*\(' \
     /home/*/public_html --include='*.php' | head -30

# Executables somewhere they have no business being
find /tmp /var/tmp /dev/shm -type f -executable -ls

4. Persistence mechanisms

An attacker who only has a webshell loses access when you patch. So they add a way back in.

# Every user's crontab
for u in $(cut -f1 -d: /etc/passwd); do
  echo "--- $u"; crontab -l -u "$u" 2>/dev/null
done
ls -la /etc/cron.*/ /var/spool/cron/

# Unexpected SSH keys — check every account, not just root
find /home /root -name authorized_keys -exec ls -l {} \; -exec cat {} \;

# Accounts with UID 0 other than root
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Accounts with a login shell that shouldn't have one
awk -F: '$7 !~ /(nologin|false)$/ {print $1, $7}' /etc/passwd

# Recently added or modified systemd units
find /etc/systemd/system /usr/lib/systemd/system -mtime -14 -ls

5. Log gaps and quiet tampering

# Successful SSH logins — anything unfamiliar?
grep 'Accepted' /var/log/secure | tail -40
last -20

# A truncated or unusually small log is a signal in itself
ls -la /var/log/{secure,auth.log,wtmp,btmp}

# Verify system binaries against package checksums
rpm -Va --nomtime --nofiledigest | grep -E '^..5'   # RHEL family
debsums -c                                          # Debian family

That verification step is genuinely valuable. If ps, ls, netstat or find fails a checksum, you're dealing with a rootkit and everything the server has told you so far is unreliable.

If you find something: the order matters

This is where most people go wrong. The instinct is to delete the malicious file immediately. Resist it for ten minutes.

1. Preserve before you change. Snapshot the VM if you can. Otherwise capture the volatile state — running processes, network connections, open files, logged-in users — to a file off the server. It disappears the moment you reboot.
2. Contain, don't destroy. Suspend the affected account, or firewall the server down to your own management IP. This stops the ongoing harm — spam, attacks on others — without wiping the evidence.
3. Scope it. How far did they get? One account, or root? On a shared server, check whether the same webshell signature appears under other users. Establish the earliest timestamp you can find — that's when it started, and anything after it is suspect.
4. Find the entry point. Correlate the webshell's creation time with the access logs for that minute. It's usually an outdated plugin, a file upload with no validation, or reused credentials. If you skip this step, you will do the whole thing again next month.
5. Eradicate. Prefer restoring from a backup taken before the earliest indicator over hand-cleaning. Hand-cleaning means being sure you found every file, and you can't be sure.
6. Rotate every credential the compromised context could reach: SSH keys, panel logins, database passwords, API keys, mail passwords. Assume all of them were read.
7. Close the hole, then restore service — patch the vulnerable component, fix the permissions, add the WAF rule.
8. Watch. Attackers come back to check on their access. Monitor closely for a fortnight; a returning connection attempt tells you whether you actually got everything.

D=/root/ir-$(date +%F-%H%M); mkdir -p "$D"
ps auxwwf            > "$D/processes.txt"
ss -tupn             > "$D/connections.txt"
lsof -nP             > "$D/openfiles.txt" 2>/dev/null
last -50             > "$D/logins.txt"
crontab -l -u root   > "$D/root-cron.txt" 2>/dev/null
cp /var/log/secure /var/log/messages "$D/" 2>/dev/null
tar czf "$D.tar.gz" "$D" && echo "captured: $D.tar.gz"
# Now copy that archive OFF the server before you continue.

When to rebuild instead of clean

Some findings mean cleaning is no longer a defensible choice. Rebuild if:

• Root was compromised. With root, anything can be modified — including the tools you'd use to verify the cleanup.
• System binaries fail checksum verification. You can't trust the server's own account of itself.
• You can't establish the entry point. An unknown hole is an open hole.
• Kernel modules were loaded that you can't account for.

Rebuilding is slower and nobody enjoys telling a customer. It's still cheaper than discovering three weeks later that the backdoor survived the cleanup.

Making the next one visible sooner

• File integrity monitoring (AIDE, Tripwire, or the panel's own scanner) alerting on changes under web roots outside a deploy window.
• Logs shipped off the box in real time. Local logs are the first thing edited.
• Outbound mail rate limits per account. Turns a spam run from a blacklisting into an alert.
• Egress filtering. Web servers rarely need to make arbitrary outbound connections; blocking that breaks most droppers and miners.
• A maintained inventory of CMS and plugin versions, so "which sites are vulnerable to this new CVE" is a query, not a project.

The pattern worth internalising: the compromise you notice is rarely the compromise that matters. Defacement, blacklisting and ransom notes are the visible end of something that started quietly weeks earlier. Look for the quiet part — and when you find it, take the ten minutes to preserve evidence before you start deleting. That decision is usually the difference between fixing it once and fixing it monthly.