Problem Statement

Application servers suddenly failed to create temp sockets or write pidfiles:

BASH 
touch /tmp/test.txt
touch: cannot touch '/tmp/test.txt': No space left on device

Yet df -h reported over 420 GB of free space on /dev/sda2.

Inspecting Inode Allocation

Running df -i immediately clarified the outage:

TEXT 
Filesystem      Inodes     IUsed    IFree  IUse% Mounted on
/dev/sda2     32768000  32768000        0   100% /

The ext4 inode table was 100% saturated. Any new file creation was rejected at the VFS layer.

Root Cause & Fast Deletion

A legacy PHP session cleanup cronjob had failed due to a malformed syntax in a sibling file. As a result, /tmp accumulated over 10.4 million session files.

Running rm -rf /tmp/sess_* fails with Argument list too long.

We purged the directory using directory syncing:

BASH 
mkdir /empty_dir
rsync -a --delete /empty_dir/ /tmp/

Within 4 minutes, all 10 million inodes were reclaimed.

Permanent Architecture Fixes

  • Migrated sessions out of filesystem into Redis clusters with strict TTL.
  • Mounted /tmp as tmpfs with explicit inode bounds: mount -t tmpfs -o size=2G,nr_inodes=500k tmpfs /tmp.
  • Added Prometheus alerts on node_filesystem_files_free ratio drop below 20%.