In this guide 6 sections
Disks fill when something writes faster than anyone expects: application logs, database transaction logs, old container images or backups stored on the same volume. A fixed alert at a set fullness arrives too late. Alert on how fast free space is falling and the hours left.
Find what is actually eating the space
Start from the top. Run df -h to see which volume is full, then du -xh --max-depth=1 on that mount to find the heavy folder. Repeat one level down until a single name stands out from the rest.
Deleted files can still hold space. A process that keeps a log open after the file is removed continues to use the blocks, which lsof +L1 will list. Restart that process. The space returns at once.
Inodes are a second limit. A folder holding millions of tiny PHP session files can exhaust inodes while df still shows free gigabytes, and df -i reveals it.
Logs that grow without a ceiling
Application logs are the usual cause. A loop that retries a failed call writes thousands of lines a minute, a debug flag left on after a release does the same quietly. Nginx access logs on a busy site grow all day long.
Rotation fixes this. Configure logrotate or the journal size limit so each log has a maximum size, a number of kept files and compression. Ship logs to a central store so the server keeps only a few days.
Then ask why the volume of logs jumped. A sudden rise in errors is a symptom worth reading.
- Application and framework error logs
- Nginx or Apache access and error logs
- System journal and syslog
- Debug output left on after a release
Database logs and temporary files
Databases keep their own logs. PostgreSQL holds write-ahead log segments until archiving or a replica has read them. A stalled replica makes the WAL folder swell, while MySQL keeps binary logs for as long as its expiry setting allows.
Large sorts and queries spill into temporary files. One report that joins two huge tables can write gigabytes to disk in minutes and then vanish, leaving no trace except the outage.
Check replication lag and the age of the oldest unarchived segment. Both give early warning. Set a cap on temp_file_limit in PostgreSQL or tmp_table_size in MySQL, so one careless report cannot take the whole volume with it.
Container images and backups kept on the same volume
Docker keeps every image layer from every deployment. After a hundred releases, the old layers, stopped containers and unused volumes can outweigh the live application, and a scheduled prune with a retention rule clears them.
Local backups cause a quieter problem. A nightly dump to the same disk grows by one database size each day, and a failed clean-up job leaves thirty copies behind. A backup on the volume it protects is no protection when that disk fails.
Send backups to object storage such as S3 with a lifecycle rule. Keep a short local copy only for fast restores.
- Old image layers and stopped containers
- Unused named volumes
- Database dumps left in home folders
- Core dumps from crashed processes
Alert on growth rate, not on a fixed percentage
A threshold at a set level fails in two ways. On a large volume it fires too late, since a small slice still equals hours of writing. On a small volume it fires constantly until everyone learns to ignore it.
Track growth instead. Prometheus offers predict_linear, which estimates when free space reaches zero from the last few hours of data. Alert when the estimate is under twenty-four hours and page someone when it is under four.
Next step this week: pick your busiest server and record its free space three times a day. The slope you see is the number to alert on.
Disk space triage sheet
Use this at an incident or a weekly review. A table holds volume, size, free space, growth per day and projected full date. Beneath it sit a short command list, one line per usual cause (logs, database logs, images, backups) and a box for the fix applied.
Open a blank worksheet to printQuestions owners ask
Why does df show a full disk when my folders add up to less?
Deleted files held open by a running process still use blocks. List them with lsof +L1 and restart the process. Reserved filesystem blocks and mounted volumes can also account for the gap.
Is it safe to delete log files on a live server?
Truncate rather than delete a log that a process holds open, for example with the truncate command. Deleting the file leaves the process writing to hidden blocks, and the space does not return.
How much free space should I keep?
There is no single figure. Keep enough for several hours of your fastest likely growth plus room for a database to rebuild an index. Your growth measurements tell you the right margin.