排查CentOS磁盘占用

TL;DR

一个服务器磁盘占用多了，本文是排查磁盘占用和清理的经验。

有用的命令：

df -h 磁盘当前总体占用情况

du -h -t 1G / 从根目录查找大于1G的目录或文件

du -h -s /usr/* | sort -nr 显示/usr/目录下各子目录的大小

排查和清理过程

1、先看看当前磁盘总体情况 df -h

[root@VM_0_11_centos /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        50G   44G  3.2G  94% /
devtmpfs        909M     0  909M   0% /dev
tmpfs           920M   24K  920M   1% /dev/shm
tmpfs           920M  544K  919M   1% /run
tmpfs           920M     0  920M   0% /sys/fs/cgroup
tmpfs           184M     0  184M   0% /run/user/0

可以看到目前磁盘已经使用了94%了，需要关注并尝试清理了。磁盘满了会导致主机上的应用故障。

2、然后在从根目录排查哪个目录、文件占用磁盘空间比较多 du -h -s | sort -nr

[root@VM_0_11_centos /]# du -s -h /* | sort -nr
du: cannot access ‘/proc/1807/task/1807/fd/4’: No such file or directory
du: cannot access ‘/proc/1807/task/1807/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/1807/fd/4’: No such file or directory
du: cannot access ‘/proc/1807/fdinfo/4’: No such file or directory
573M    /home
544K    /run
503M    /data
159M    /opt
156K    /tmp
108M    /boot
57M     /etc
26G     /usr
16K     /lost+found
11G     /var
4.1G    /root
4.0K    /srv
4.0K    /mnt
4.0K    /media
2.1G    /swapfile
0       /sys
0       /sbin
0       /proc
0       /lib64
0       /lib
0       /dev
0       /bin

占用最多的是/usr， 那么继续从这个目录向下排查：

[root@VM_0_11_centos /]# du -h -s /usr/* | sort -nr
816M    /usr/hbase-2.3.7
704M    /usr/hbase-2.2.4
657M    /usr/bin
541M    /usr/java
504M    /usr/lib
465M    /usr/apache-phoenix-5.0.0-HBase-2.0-bin
452M    /usr/sbin
424M    /usr/lib64
417M    /usr/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz
415M    /usr/hbase-1.3.6
333M    /usr/share
261M    /usr/hbase-2.3.7-bin.tar.gz
220M    /usr/phoenix-hbase-2.3-5.1.2-bin
214M    /usr/hbase-2.2.4-bin.tar.gz
194M    /usr/phoenix-hbase-2.3-5.1.2-bin.tar.gz
193M    /usr/kafka_2.12-1.0.0
152M    /usr/redis-6.0.3
152K    /usr/zooData
107M    /usr/hbase-1.3.6-bin.tar.gz
106M    /usr/src
55M     /usr/libexec
53M     /usr/rocketmq-4.8.0
43M     /usr/kafka_2.12-1.0.0.tgz
42M     /usr/apache-zookeeper-3.6.3-bin
22M     /usr/include
17G     /usr/local
12M     /usr/apache-zookeeper-3.6.3-bin.tar.gz
8.0K    /usr/redis-script
4.0K    /usr/games
4.0K    /usr/etc
2.2M    /usr/redis-6.0.3.tar.gz
1.2G    /usr/prometheus
1.1G    /usr/mysql-5.7
0       /usr/tmp

[root@VM_0_11_centos /]# du -h -s /usr/local/* | sort -nr
731M    /usr/local/qcloud
96K     /usr/local/lib64
46M     /usr/local/bin
16G     /usr/local/openresty
4.0K    /usr/local/src
4.0K    /usr/local/sbin
4.0K    /usr/local/libexec
4.0K    /usr/local/lib
4.0K    /usr/local/include
4.0K    /usr/local/games
4.0K    /usr/local/etc
1.2M    /usr/local/share
0       /usr/local/yd.socket.server

[root@VM_0_11_centos /]# du -h -s /usr/local/openresty/nginx/* | sort -nr
84K     /usr/local/openresty/nginx/conf
60K     /usr/local/openresty/nginx/html
32M     /usr/local/openresty/nginx/sbin
16K     /usr/local/openresty/nginx/tapset
16G     /usr/local/openresty/nginx/logs
4.0K    /usr/local/openresty/nginx/uwsgi_temp
4.0K    /usr/local/openresty/nginx/scgi_temp
4.0K    /usr/local/openresty/nginx/proxy_temp
4.0K    /usr/local/openresty/nginx/fastcgi_temp
4.0K    /usr/local/openresty/nginx/client_body_temp

[root@VM_0_11_centos /]# du -h -s /usr/local/openresty/nginx/logs/* | sort -nr
190M    /usr/local/openresty/nginx/logs/access.log
16G     /usr/local/openresty/nginx/logs/error.log
4.0K    /usr/local/openresty/nginx/logs/nginx.pid

找到原因了，nginx的error.log文件很久没清理了。

3、清空文件

cat /dev/null > /usr/local/openresty/nginx/logs/error.log

再次查看各个文件大小

[root@VM_0_11_centos /]# du -h -t 1G /
1.2G    /usr/prometheus
1.1G    /usr/mysql-5.7
11G     /usr
4.0G    /var/log/journal/0ea734564f9a4e2881b866b82d679dfc
4.0G    /var/log/journal
4.6G    /var/log
4.9G    /var/lib/mysql/test_perf_db
5.2G    /var/lib/mysql
5.7G    /var/lib
11G     /var
3.6G    /root/logs/rocketmqlogs
3.7G    /root/logs
4.1G    /root
29G     /

类似的，再去清理rocketmqlogs