2019-01-31 MySQL 的一次启动失败的问题定位
2019-01-31 本文已影响0人
浪飞人海
MySQL 的一次启动失败的问题定位
问题现象
复现步骤如下：
1）执行重启mysql服务的命令： service mysql restart
2）几秒钟后，出现提示：
$ sudo service mysql start
[sudo] password for lintong:
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
3）根据提示：输入systemctl status mysql.service,得到如下的提示信息：
$ systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since 四 2019-01-31 16:22:35 CST; 23s ago
Process: 28233 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
Process: 28217 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 28233 (code=exited, status=1/FAILURE); : 28234 (mysql-systemd-s)
Tasks: 2
Memory: 3.7M
CPU: 344ms
CGroup: /system.slice/mysql.service
└─control
├─28234 /bin/bash /usr/share/mysql/mysql-systemd-start post
└─29680 sleep 1
到这里还并不能定位到问题。
4）继续按照提示，输入命令journalctl -xe，输出如下的信息：
$ sudo journalctl -xe
1月 31 16:23:02 Z840 kernel: keepalived[29923]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:05 Z840 systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
1月 31 16:23:05 Z840 systemd[1]: mysql.service: Unit entered failed state.
1月 31 16:23:05 Z840 systemd[1]: mysql.service: Failed with result 'exit-code'.
1月 31 16:23:05 Z840 systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
1月 31 16:23:05 Z840 systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
1月 31 16:23:05 Z840 systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30151/status" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=126
1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=0
1月 31 16:23:05 Z840 audit[30151]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/30151/status" pid=30151 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=126 ouid=126
1月 31 16:23:06 Z840 kernel: show_signal_msg: 29 callbacks suppressed
1月 31 16:23:06 Z840 kernel: keepalived[30201]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:06 Z840 kernel: keepalived[30213]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:06 Z840 kernel: keepalived[30218]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:06 Z840 kernel: keepalived[30230]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:06 Z840 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
1月 31 16:23:06 Z840 kernel: keepalived[30238]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:06 Z840 kernel: keepalived[30245]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:07 Z840 kernel: keepalived[30256]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:07 Z840 kernel: keepalived[30261]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:07 Z840 kernel: keepalived[30274]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:07 Z840 kernel: keepalived[30280]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:11 Z840 kernel: show_signal_msg: 27 callbacks suppressed
1月 31 16:23:11 Z840 kernel: keepalived[30521]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:11 Z840 kernel: keepalived[30534]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:11 Z840 kernel: keepalived[30538]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:11 Z840 kernel: keepalived[30546]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:11 Z840 kernel: keepalived[30553]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:12 Z840 kernel: keepalived[30565]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:12 Z840 kernel: keepalived[30578]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:12 Z840 kernel: keepalived[30582]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:12 Z840 kernel: keepalived[30591]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:12 Z840 kernel: keepalived[30597]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:16 Z840 kernel: show_signal_msg: 25 callbacks suppressed
1月 31 16:23:16 Z840 kernel: keepalived[30833]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:16 Z840 kernel: keepalived[30846]: segfault at 0 ip 0000000000418218 sp 00007fff07055190 error 4 in keepalived[400000+49000]
1月 31 16:23:16 Z840 sudo[30850]: lintong : TTY=pts/20 ; PWD=/home/lintong ; USER=root ; COMMAND=/bin/journalctl -xe
1月 31 16:23:16 Z840 sudo[30850]: pam_unix(sudo:session): session opened for user root by lintong(uid=0)
5）MySQL的运行日志，在/var/log/mysql/下能找到，但是发现大小为空。
# lintong @ Z840 in / [17:30:27]
$ sudo ls -al /var/log/mysql/
[sudo] password for lintong:
total 64
drwxr-x--- 2 mysql adm 4096 1月 31 16:17 .
drwxrwxr-x 20 root syslog 4096 1月 31 16:31 ..
-rw-r----- 1 mysql adm 0 1月 31 17:30 error.log
-rw-r----- 1 mysql adm 20 1月 30 07:35 error.log.1.gz
-rw-r----- 1 mysql adm 986 1月 29 15:40 error.log.2.gz
-rw-r----- 1 mysql adm 618 1月 29 00:31 error.log.3.gz
-rw-r----- 1 mysql adm 20 1月 27 07:35 error.log.4.gz
-rw-r----- 1 mysql adm 20 1月 26 07:35 error.log.5.gz
-rw-r----- 1 mysql adm 2258 1月 25 17:47 error.log.6.gz
-rw-r----- 1 mysql adm 20 1月 25 06:40 error.log.7.gz
-rw-r----- 1 mysql adm 0 1月 31 07:35 general.log
-rw-r----- 1 mysql adm 20 1月 30 07:35 general.log.1.gz
-rw-r----- 1 mysql adm 20 1月 29 07:35 general.log.2.gz
-rw-r----- 1 mysql adm 20 1月 28 07:35 general.log.3.gz
-rw-r----- 1 mysql adm 20 1月 27 07:35 general.log.4.gz
-rw-r----- 1 mysql adm 20 1月 26 07:35 general.log.5.gz
-rw-r----- 1 mysql adm 20 1月 25 07:35 general.log.6.gz
-rw-r----- 1 mysql adm 20 1月 24 07:35 general.log.7.gz
问题定位
采用技术问题的七步解决法，优先到Stack Overflow中搜索，发现有一个与我遇到的问题非常相似：
MySQL failed to restart on Ubuntu 16.04
但是在journalctl -xe 日志存在差异，因为事先已经知道目前的机器内存不足，因此怀疑是内存不足引起的。
正好在后面的回答中，也有类似的出现的情况：
问题根因：空间不足。
问题解决
清理磁盘的空间。
