Finding botnets on servers - introducing "shtracer"

Updated May 2, 2018 - note new references to Laskowski-Tech solutions at the end.

This page is specifically intended for the administrator of a computer system, because we describe tactics and methods that ONLY the system administrator can do.

If you are not the administrator of the computer system corresponding to the IP address listed by the CBL, this page is not for you, and you should bring it to the attention of the system administrator. A system administrator generally has the ability to become "superuser" on the computer. If you can't do that/don't know the superuser ("root") password, you're not the system administrator. As the listings that lead you here are frequently on shared web hosting environments, you need to contact your web hosting provider.

The most difficult part of removing a botnet is "where is it?". You can't fix it if you can't find it. This is particularly hard in shared hosting environments where there can be hundreds or thousands of separate virtual hosts, all with different software and different levels of knowledge in keeping them secure.

Typically, a botnet attempts to connect to "Command and Control" (C&C) sites and retrieve instructions on how to spam, DDOS, steal private information or otherwise do malicious things.

Anti-botnet researchers are often able, with the assistance of domain registrars, to seize one or more domain names that the botnet uses to find the C&C. Such researchers often stand up a computer (called a sinkhole) using the C&C domain name. When the botnet infection attempts to "reach out" for instructions it hits the sinkhole instead. The sinkhole can then identify the botnet type (from the domain used), the IP address of the botnet infection, timestamps and other information. The CBL collates lists from various researchers (including ourselves) and produces listings from them. When you do a lookup of an IP listing and see it's a botnet, there's usually a table of explicit detail about the C&C connection.

Armed with the C&C connection information, the assumption is that the administrator can find and eradicate the malware. If you can find the connection in a suitably detailed log (from firewall or other), or have advanced administration tools (managed switches and the like), you can look for new C&C connections in real time and thus find the infection.

Unfortunately, even with the C&C information, it's not always easy for an administrator to find the infection, especially in less sophisticated environments with fewer advanced tools.

Generally speaking, there's one of two scenarios at play:

  1. The IP corresponds to a single machine. Now you have to find which process/program is issuing the C&C connections, or
  2. The IP corresponds to a NAT or PAT gateway, where first you have to identify which machine is issuing them, and then go back to scenario 1.

The CBL has written a series of pages on how to find an infected machine, for example How to find BOTs in a LAN. But we've been lacking for (1). We've decided to try to address that and have written a tool (in perl) which works on most Linuxes, and probably on FreeBSD and other UNIX-work-alikes.

In sophisticated environents with advanced tools and expert administrators, finding the process making particular connection is usually fairly easy using this like iptables rules or snort. But we decided to build something that any administrator could run without advanced tools or expertise.

One of the simplest ways of checking for outbound Internet connections is via using the standard Linux tool "iptables". To use iptables, change the string "%%SINKHOLE%%" to the IP address of the sinkhole given in the CBL lookup, and you can see these log records in the system logs (often /var/log/kern.log).

sudo iptables -I OUTPUT -m state -p tcp --state NEW -d %%SINKHOLE%% -m limit --limit 10/m --limit-burst 10 -j LOG --log-uid --log-prefix "Outbound Connection:  "

The drawbacks to using iptables is that iptables doesn't log processid or process name, won't work on FreeBSD and derivitives etc. This isn't enough in large systems doing lots of networking activity.

The CBL has written a Perl Script that specifically looks for C&C connections on Linux and Linux-like systems (like FreeBSD or MacOS), and reports detail about them.

As configured, tool does a query every .01 seconds looking for connections to the C&C IP address. If it finds one it will display information about the connection, including process ID. As the connections happen very fast, the process ID may not live long enough to be able to identify the process by waiting for you to notice this and do a "ps". As configured, it will attempt to use /proc (if available) to tell you more information about the process (command, file, command line). If /proc is not available, it will use the "lsof" tool to tell you the name of the process and other details, which should allow you to find it on disk.

You may need to run this program as root for hours before it will show these connections. We recommend running it for at least 12 hours before assuming that such connections are not happening.

This script can be picked up here. There is an newer version taking advantage of specific features of Linux, you can try here Instructions on how to use it are contained in the script. Please remember that it's beta software, and suggestions/improvements are welcome.

Two other valuable approaches, both from

  1. EITest Cleanup Part 1
  2. EITest Cleanup Part 2

Part 1 is a set of scripts similar in method to our script, but perhaps better to CentOS versions of Linux. Part 2 is a different approach using Yara and ClamAV. Part 2 will catch all "known" versions of EITest resident at once on a multi-host server (if run by root) by matching Yara signatures in the files, and doesn't require the script "catch the EITest infection in the act". Under most circumstances it should be far more complete, reliable and faster. Part 1 works if there's a previously unknown version of EITest.

If you have any suggestions on how to improve, please let us know.