如何检查Linux服务器是不是被黑了

1 - Check the last users connection on the host

From console digit:

# last

This command will show you all user that logged successfully in the host. 

This is a good point to start to understand if somebody get an unauthorised access on your server.

This is one example of the command :

root@servername~# last 

root pts/0 10.10.50.3 Mon Jan 30 13:03:45 2017 still logged in

root pts/0 10.10.50.3 Mon Jan 30 13:00:27 2017 - Mon Jan 30 13:03:44 2017 (00:03)

root pts/0 10.10.50.3 Mon Jan 30 12:50:42 2017 - Mon Jan 30 13:00:26 2017 (00:09)

root pts/0 10.10.50.3 Fri Jan 27 09:17:39 2017 - Fri Jan 27 16:44:53 2017 (07:27)

2 – Check the running processes with TOP

Personally I prefer top to “ps aux” just because top will provide you some extra information that can be useful.

With top you can verify if some application are using more resource than usual.

With this command you can also find other useful information, like free Ram, average load, number of task, CPU load ect.

# top

top - 11:33:21 up 21 days, 41 min, 1 user, load average: 0.02, 0.02, 0.05

Tasks: 90 total, 2 running, 88 sleeping, 0 stopped, 0 zombie

%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

KiB Mem : 6808648 total, 4629072 free, 306808 used, 1872768 buff/cache

KiB Swap: 0 total, 0 free, 0 used. 5962784 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1 root 20 0 125104 3648 2408 S 0.0 0.1 1:40.58 systemd

2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd

3 root 20 0 0 0 0 S 0.0 0.0 0:00.89 ksoftirqd/0

5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H

7 root rt 0 0 0 0 S 0.0 0.0 0:01.84 migration/0

8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh

As default TOP will update himself every second with the latest information, but will show you only a limited number of application

If you’d like see all application are running the best is to add more:

# top | more

Optional: Kill a process

If you need to kill a process you can do it in 2 ways

The easly one is to digit

# pidof processname

Example to kill all process of apache

# pidof apache

However you can also kill a specific pid digit the command “kill” followed to the pid number that you can find in “Top” or “ps aux”

Example: 

# kill 3445

3 - Check SSH attempt connections:

Another step is to check the ssh logs to understand is somebody is trying to get access to the server, 

You can check the access log to the server ( SSH ) in this way

From console digit:

# tail -n 500 /var/log/secure | grep ‘sshd’ ( if you are using a Centos/RedHat distribution based )

# tail -n 500 /var/log/auth.log | grep sshd ( if you are using a Debian distribution based )

This is one example:

root@servername:~# tail -n 1000 /var/log/auth.log | grep sshd

Jan 30 13:24:12 servername sshd[17655]: pam_unix(sshd:session): session opened for user root by (uid=0)

Jan 30 13:26:43 servername sshd[17796]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.50.3 user=root

Jan 30 13:26:45 servername sshd[17796]: Failed password for root from 10.0.50.3 port 14238 ssh2

Jan 30 13:26:51 servername sshd[17804]: Accepted password for root from 10.0.50.3 port 14332 ssh2

This command will show you the log ( from the last 500 lines ) of all the attempt to get into the ssh server (failed or authorized) with important information such as the username was trying to get 

access source IP

Note:

If you need to read backward the log you need to increase the number of line maybe to 1000 or more depend of the server use because this logfile store all access to the server ( FTP SSH Webmin ldap, and other… ) also after X time the server create a new file to keep the file size small, that means maybe you have to open some old logs.

3a Optional: Last command by user

In case someone get access to the server probably he sent some command trough the ssh.

The follow command will show the last 100 line of command digit on the console for the current user

tail -n 100 ~/.bash_history | more

you can use nano instead of tail to read all the log.

nano ~/.bash_history | more

if you’d like you can check also command digit from other user. To be able to do that you have to visit they own home folder, generally is /home

Example:

tail -n 100 /home/username/.bash_history | more

4 - Netstat :Verify the current connections from and to the server

Another step is to verify the connection to your server with netstat

From console digit:

# netstat | more

this command is will show you 2 part the first is “Active Internet connections” (w/o servers) and the second is “Active UNIX domain sockets (w/o servers)”

check both carefully because if you got a malicious script is trying to sending spam mail or try to attach other server you can easily find here

The point of the situation:

After this 4 step if you server has been hacked you should find something, but to be sure is better do some extra checks

5 - Check IP Ban / Blacklist for the Public IP Address

The best web is google a bit and find a couple website that allow you to verify if your ip is banned.

One of the website im using is: http://whatismyipaddress.com/blacklist-check

If your public ip is banned, probably that means that your server was doing something wrong like SSH Bruten force, email spam, DDos.

However that don’t means that you server has been hacked but maybe one of the website or web application has been hacked or do something wrong.

That means that you should check the website

6 - Crontab: scheduled jobs

Another best practice is to check the cronjobs ( for all users )

Maybe a malicious script or application could be seen here

# cd /var/spool/cron/crontabs/ && grep . * ( for Debian distribution based )

# cd /var/spool/cron && grep . * ( for Centos/RedHat distribution based )

this command will show you all the cron jobs for all users, that can show you some more information to understand if there is some malicious script is running at the reboot or every X time

7 - Antivirus scan

A good practice is to install an antivirus and run a scan when you think that the system can have virus.

To install the AV on

debian:

Apt-get install clamav-scan

Redhat:

yum install clamav-server clamav-data clamav-update clamav-filesystem clamav clamav-scanner-systemd clamav-devel clamav-lib clamav-server-systemd

now you need to update the AV:

# freshclam

This command will start the update of the virus definition, soon the update has been completed you can run the scan typing

# clamscan –r / | grep FOUND >> /var/log/av/AvReport.txt

Adding the second part eventually virus report will be logged into the file AvReport.txt

NOTE: I suggest to avoid to setup a schedule or run the AV in realtime, not just for an eventual performance degradation but because the AV if find a false-positive will be delete automaticly a file and I don’t think this is a good idea in some kind of servers