There are several ways you might discover a break-in:
There are a variety of commands that you can use to discover a break-in, including lsof , top , ps , and netstat . There are also several packages that you can use, including Tripwire, that are described elsewhere in this book. Use these tools on a regular basis, but use them sporadically as well. This introduces an element of randomness that can keep perpetrators from being able to cover their tracks. This principle is a standard of operations security: try to be unpredictable. 22.2.1 Catching One in the ActThe easiest way to catch an intruder is by looking for events that are out of the ordinary. For example:
22.2.1.1 Monitoring commandsUnix provides a number of commands to help you figure out who is doing what on your system. The finger , users , whodo , w , and who commands all display lists of the users who are currently logged in. The ps and w commands help you determine what any user is doing at any given time, ps displays a more comprehensive report, and w displays an easy-to-read summary. The netstat command can be used to check on current network connections and activity. If you are a system administrator, you should be in the habit of issuing these commands frequently to monitor user activity. After a while, you will begin to associate certain users with certain commands. Then, when something out of the ordinary happens, you will have cause to take a closer look. Be aware, however, that all of these commands can be "fooled" by computer intruders with sufficient expertise. For example, w , users , and finger all check the /etc/utmp file to determine who is currently logged into the computer. If an intruder erases or changes his entry in this file, these commands will not report the intruder's presence. Also, some systems fail to update this file, and some window systems do not properly update it with new entries, so even when the file is protected, it may not have accurate information. As the ps command actually examines the kernel's process table, it is more resistant to subversion than the commands that examine the /etc/utmp file. However, an intruder who also has attained superuser access on your system can modify the ps command or the code in the system calls it uses so that it won't print her processes. The system kernel can be modified so that it hides commands or network connections by an attacker. Furthermore, any process can modify its argv arguments, allowing it to display whatever it wishes in the output of the ps command. If you don't believe what these commands are printing, you might be right! 22.2.1.2 Other tip-offsThere are many other tip-offs that an intruder might be logged onto your system. For example, you may discover that shells are running on terminals that no one seems to be logged into at the moment. You may discover open network connections to machines you do not recognize. Running processes may be reported by some programs but not others. Be suspicious and nosy. 22.2.2 What to Do When You Catch SomebodyYou have a number of choices when you discover an intruder on your system:
If you are most inclined towards option #1, you probably should reread the earlier chapters of this book. Ignoring an intruder who is on your system essentially gives him free reign to do harm to you, your users, and others on the network. You may also put yourself at risk for downstream liability if the intruder causes damage at another organization and you had the chance to stop him. If you choose option #2, keep a log of everything the intruder sends back to you, then decide whether to pursue one of the other options. You should also be exceedingly careful if you pursue this approach: an attacker who has achieved superuser access on your computer may not take well to being contacted, and may respond by attempting to wipe out your computer system with a hastily-typed rm -rf / or similar command. Options #3 and #4 are discussed in Section 22.2.5 and Section 22.2.7, which follow. Option #5 is covered in Chapter 23. 22.2.3 Contacting the IntruderIf somebody is logged into your computer without authorization, your first reaction may be to contact him. However, you should be extremely careful if you pursue this course of action. Some intruders are malicious in intent, or extremely paranoid about being caught. If you contact them, they may react by trying to delete everything on your computer to hide their activities. If you try to contact an intruder, be certain you have a set of extremely current backups ! If you intend to trace the intruder, you should trace the intruder first , before you attempt to contact the individual. [2] This is because it is much easier to trace an active network connection than one that has been disconnected.
After the trace, you may want to try to communicate with the intruder using the write or talk programs. If the intruder is connected to your computer by a physical terminal, you might want to walk over to that terminal and confront the person directly (then again, you might not!). 22.2.4 Monitoring the IntruderYou may wish to monitor the intruder's actions to figure out what he is doing. This will give you an idea of whether the intruder is modifying your accounting database or simply rummaging around through your users' email. However, keep in mind that you don't know how long this intruder has been on your system, so all you can really monitor is what the intruder does next . There are a variety of means that you can use to monitor the intruder's actions. The simplest way is to use programs such as ps or lastcomm to see which processes the intruder is using. If the intruder is logged in over a network connection, you can use a packet monitor such as tcpdump , ethereal , or snoop either to display the user's packets or to record them in a file. These programs allow monitoring of all packets received and transmitted by your computer. (When using tcpdump , be sure to specify the flag -s 4096 so that entire IP packets are recorded; the default is to record only the first 68 bytes.) If your computer is attached to a hub and has an appropriate Ethernet interface card, it can record packets sent to other computers as well. In a network that is based on hubs, you will be able to see all of the traffic that is transmitted between other computers; in a switched environment, you will only be able to eavesdrop on broadcast messages. If your intruder is logged on through a modem or serial port that is connected directly to your computer, there are several programs that you can use to monitor the intruder's actions, including ttywatch (available for Linux and Solaris), conserver (http://www.conserver.com/), rtty , and ser2net . These programs can give you a detailed, byte-by-byte account of the information that is sent over one or more serial ports. In many cases, they can also monitor pseudo- tty s, which is valuable when the attacker has connected over the network using an encrypted protocol such as SSH. Example 22-1 shows output provided by the snoop program. In this case, an email message was intercepted as it was sent from asy8.vineyard.net to the computer next. Example 22-2 shows traffic displayed by the tcpdump program. In this case, the traffic displayed is web traffic. As the examples show, these utilities can give you a detailed view of what people on your system are doing on the network, but they also have a great potential for abuse. Example 22-1. Traffic collected and displayed by the snoop program# snoop asy8.vineyard.net -> next SMTP C port=1974 asy8.vineyard.net -> next SMTP C port=1974 MAIL FROM:<dfddf@vin next -> asy8.vineyard.net SMTP R port=1974 250 <dfddf@vineyard. asy8.vineyard.net -> next SMTP C port=1974 asy8.vineyard.net -> next SMTP C port=1974 RCPT TO:<vdsalaw@ix. next -> asy8.vineyard.net SMTP R port=1974 250 <vdsalaw@ix.netc asy8.vineyard.net -> next SMTP C port=1974 asy8.vineyard.net -> next SMTP C port=1974 DATA\r\n next -> asy8.vineyard.net SMTP R port=1974 354 Enter mail, end Example 22-2. Traffic displayed by the tcpdump programr2# tcpdump tcpdump: listening on dc0 22:27:12.387205 lsanca1-ar19-4-46-094-068.lsanca1.dsl-verizon.net.54487 > r2.http: . ack 2742304626 win 11040 <nop,nop,timestamp 340262 71073569> (DF) 22:27:12.510148 TYOnni-09p204.ppp12.odn.ad.jp.3463 > sdsl-64-7-15-236.dsl.bos.megapath. net.http: . ack 2590430033 win 16019 22:27:12.585941 lsanca1-ar19-4-46-094-068.lsanca1.dsl-verizon.net.54487 > r2.http: . ack 2649 win 8392 <nop,nop,timestamp 340262 71073569> (DF) 22:27:12.760694 66.113.136.247.43790 > r2.smtp: S 3228789179:3228789179(0) win 5840 <mss 1460,sackOK,timestamp 1149898309 0,nop,wscale 0> (DF) 22:27:12.988674 lsanca1-ar19-4-46-094-068.lsanca1.dsl-verizon.net.54487 > r2.http: . ack 4097 win 6944 <nop,nop,timestamp 340263 71073569> (DF) 22:27:13.187460 lsanca1-ar19-4-46-094-068.lsanca1.dsl-verizon.net.54487 > r2.http: . ack 6745 win 4296 <nop,nop,timestamp 340263 71073569> (DF) 22:27:13.395029 r2.1328 > 192.5.6.32.domain: 33777 PTR? 236.15.7.64.in-addr.arpa. (42) 22:27:13.416112 192.5.6.32.domain > r2.1328: 33777- 0/2/0 (90) (DF) In addition to the tools mentioned here, there are many commercial monitoring tools. Some of these tools are packet-based. Others can reassemble entire TCP/IP sessions and can pass reassembled traffic through sophisticated protocol analyzers and parsers. Be advised that it may be illegal to monitor the intruder unless you have an appropriate banner telling all users that they may be monitored . You should be careful with the tools that you install on your system, as these tools can be used against you to monitor your monitoring. If possible, you may wish to use network-monitoring tools on a computer other than the one that has been potentially compromised: doing so lessens the chance of being discovered by the intruder. 22.2.5 Tracing a ConnectionThe ps , w , and who commands all report the terminals to which each user (or each process) is attached. Terminal names like /dev/tty01 may be abbreviated as tty01 or even 01 . Generally , names like tty01 , ttya , or tty4a represent physical serial lines, while names that contain the letters p , q , or r (such as ttyp1 ) refer to network connections (virtual tty s, also called pseudo-terminals or pty s). If the intruder has called your computer by telephone, your options will depend on the services offered by your telephone company. In general, you can configure a telephone for CALLER*ID (CNID) service, but you must have a modem that is configured to receive Caller-ID information, and your operating system must capture this information. An alternative is to use a special call recorder that records the Caller-ID information for all telephone calls received and prints them on a roll of paper. Many telephone companies offer CALL-TRACE services that can be used to trace calls on an as-needed basis, but these services will generally trace only the last call. If you are receiving constant attacks, you should contact law enforcement officials to have a general trace and trap installed on your telephone line. If the intruder is logged in over the network, you can use the who command to determine quickly the name of the computer that the person may have used to originate the connection: % who orpheus console Jul 16 16:01 root tty01 Jul 15 20:32 jason ttyp1 Jul 16 18:43 (robot.ocp.com) devon ttyp2 Jul 16 04:33 (next.cambridge.m) % In this example, the user orpheus is logged in at the console, user root is logged on at tty01 (a terminal connected by a serial line), and jason and devon are both logged in over the network: jason from robot.ocp.com , and devon from next.cambridge.ma.us . Some versions of the who command (such as the one above) display only the first 16 or 32 letters of the hostname of the computer that originated the connection. (The machine name is stored in a fixed-length field in the /etc/utmp file.) To see the complete hostname, you may need to use the netstat or lsof commands (described in Chapter 24). You will also have to use netstat if the intruder has deleted or modified the /etc/utmp file to hide his presence. netstat does not reveal which network connection is associated with which user. (Of course, if you have the first 16 characters of the hostname, you should be able to figure out which is which, even if /etc/utmp has been deleted. You can still use netstat and look for connections from unfamiliar machines.) If you have the lsof command installed, this command will provide you with significantly more detail. Let's say that in this example we suspect Jason is an intruder because we know that the real Jason is at a yoga retreat in Tibet (with no terminals around). Using who and netstat , we determine that the intruder who has appropriated Jason's account is logged in remotely from the computer robot.ocp.com . We can now use the finger command to see which users are logged onto that remote computer: % finger @robot.ocp.com [robot.ocp.com] Login Name TTY Idle When olivia Dr. Olivia Layson co 12d Sun 11:59 wonder Wonder Hacker p1 Sun 14:33 % Of course, this method doesn't pin the intruder down because he may be using the remote machine only as a relay point. Indeed, in the above example, Wonder Hacker is logged into ttyp1 , which is another virtual terminal. He's probably coming from another machine, and simply using robot.ocp.com as a relay point. You would probably not see a username like Wonder Hacker . Perhaps you would see only an assorted list of apparently legitimate users and have to guess who the intruder is. Even if you did see a listing such as that, you can't assume anything about who is involved. For instance, Dr. Layson could be conducting industrial espionage on your system, using a virtual terminal (e.g., xterm ) that is not listed as a logged in session! It is also possible that the machine has been doctored in such a way that the output of the finger command does not actually reflect who is using the remote system. It's even more likely that the site's finger daemon is disabled, and you wouldn't see anything. If you have an account on the remote computer, log into it and find out who is running the rlogin or telnet commands that are coming into your computer. In any event, consider contacting the system administrator of that remote computer and alerting him to the problem. 22.2.6 How to Contact the System Administrator of a Computer You Don't KnowOften, you can't figure out the name and telephone number of the system administrator of a remote machine because Unix provides no formal mechanism for identifying such people. One good way to circumvent this problem is to contact the appropriate incident response team or the designated security person at the organization. Another way to find out the telephone number and email address of the remote administrator is to use the whois command to search the domain registration database. If your system does not have a whois command, you can simply telnet to the NIC site or use its web gateway at http://www.internic.net/cgi-bin/whois. [3] The following sections contain examples of how to find the name and phone number of a particular site administrator.
Domain name registrars are required to maintain a database of the names, addresses, and phone numbers of individuals and organizations that have registered domain names. These databases are accessible from the Unix command line using the whois command. Additional databases are available using whois as well. For example, ARIN (the American Registry for Internet Numbers) maintains a whois database that you can use to determine the individual or organization responsible for blocks of IP addresses. 22.2.6.1 Looking up information by domainHere is an example, showing how to get information about the domain purdue.edu : % whois -h whois.networksolutions.com purdue.edu The Data in the VeriSign Registrar WHOIS database is provided by VeriSign for information purposes only, and to assist persons in obtaining information about or related to a domain name registration record. VeriSign does not guarantee its accuracy. Additionally, the data may not reflect updates to billing contact information. By submitting a WHOIS query, you agree to use this Data only for lawful purposes and that under no circumstances will you use this Data to: (1) allow, enable, or otherwise support the transmission of mass unsolicited, commercial advertising or solicitations via e-mail, telephone, or facsimile; or (2) enable high volume, automated, electronic processes that apply to VeriSign (or its computer systems). The compilation, repackaging, dissemination or other use of this Data is expressly prohibited without the prior written consent of VeriSign. VeriSign reserves the right to terminate your access to the VeriSign Registrar WHOIS database in its sole discretion, including without limitation, for excessive querying of the WHOIS database or for failure to otherwise abide by this policy. VeriSign reserves the right to modify these terms at any time. By submitting this query, you agree to abide by this policy. This Registry database contains ONLY .EDU domains. The data in the EDUCAUSE Whois database is provided by EDUCAUSE for information purposes in order to assist in the process of obtaining information about or related to .edu domain registration records. The EDUCAUSE Whois database is authoritative for the .EDU domain. A Web interface for the .EDU EDUCAUSE Whois Server is available at: http://whois.educause.net By submitting a Whois query, you agree that this information will not be used to allow, enable, or otherwise support the transmission of unsolicited commercial advertising or solicitations via e-mail. You may use "%" as a wildcard in your search. For further information regarding the use of this WHOIS server, please type: help -------------------------- Domain Name: PURDUE.EDU Registrant: Purdue University Information Technology Mathematical Sciences Building West Lafayette, IN 47907-1408 UNITED STATES Contacts: Administrative Contact: Jeffrey R. Schwab Purdue University 1408 Mathematical Sciences Building West Lafayette, IN 47907-1408 UNITED STATES (765) 496-8283 jrs@purdue.edu Technical Contact: Scott M. Ballew Purdue University Mathematical Sciences Building West Lafayette, IN 47907-1408 UNITED STATES (765) 496-8232 smb@purdue.edu Name Servers: NS.PURDUE.EDU 128.210.11.5 MOE.RICE.EDU 128.42.5.4 PENDRAGON.CS.PURDUE.EDU 128.10.2.5 HARBOR.ECN.PURDUE.EDU 128.46.154.76 Domain record activated: 24-Apr-1985 Domain record last updated: 04-Jun-2002 % 22.2.6.2 Looking up information by IP addressIf you need to find out information about an IP address, say 4.46.94.68, you can use the whois or host command: % host 4.46.94.68 68.94.46.4.IN-ADDR.ARPA domain name pointer lsanca1-ar19-4-46-094-068.lsanca1.dsl- verizon.net % whois -h whois.arin.net 4.46.94.68 GENUITY (NET-GNTY-4-0) GNTY-4-0 4.0.0.0 - 4.255.255.255 GTE Intelligent Network Services (NETBLK-GTEINS-0-01) GTEINS-0-01 4.46.0.0 - 4.46.119.255 To single out one record, look it up with "!xxx", where xxx is the handle, shown in parenthesis following the name, which comes first. The ARIN Registration Services Host contains ONLY Internet Network Information: Networks, ASN's, and related POC's. Please use the whois server at rs.internic.net for DOMAIN related Information and whois.nic.mil for NIPRNET Information. [simsong@r2 ~] 334 % whois -h whois.arin.net \!net-gnty-4-0 GENUITY (NET-GNTY-4-0) 3 Van de Graaff Dr. Burlington, MA 01803 US Netname: GNTY-4-0 Netblock: 4.0.0.0 - 4.255.255.255 Maintainer: GNTY Coordinator: Soulia, Cindy (CS15-ARIN) csoulia@genuity.net 800-632-7638 Domain System inverse mapping provided by: DNSAUTH1.SYS.GTEI.NET 4.2.49.2 DNSAUTH2.SYS.GTEI.NET 4.2.49.3 DNSAUTH3.SYS.GTEI.NET 4.2.49.4 Record last updated on 02-May-2002. Database last updated on 8-Aug-2002 20:01:27 EDT. The ARIN Registration Services Host contains ONLY Internet Network Information: Networks, ASN's, and related POC's. Please use the whois server at rs.internic.net for DOMAIN related Information and whois.nic.mil for NIPRNET Information. Notice that the last whois command uses a backslash to escape the exclamation mark that is required to do a whois lookup by network "handle." 22.2.6.3 Contacting a site's ISPIf these techniques fail, you can sometimes learn the identity of the organization that controls a specific IP address by using the traceroute command: % traceroute 4.46.94.68 traceroute to 4.46.94.68 (4.46.94.68), 64 hops max, 40 byte packets 1 sdsl-64-7-15-233.dsl.bos.megapath.net (64.7.15.233) 1.115 ms 2.777 ms 1.030 ms 2 sdsl-66-80-35-1.dsl.bos.megapath.net (66.80.35.1) 9.610 ms 6.021 ms 6.110 ms 3 atm3-0.rbak1.nyc.megapath.net (66.80.133.41) 10.573 ms 10.828 ms 11.049 ms 4 ve004.bbsr1.nyc.megapath.net (66.80.132.1) 12.037 ms 12.499 ms 12.222 ms 5 gige5-1-119.ipcolo1.NewYork1.Level3.net (209.246.126.1) 10.879 ms 17.533 ms 18.242 ms 6 gigabitethernet5-2.core1.NewYork1.Level3.net (64.159.17.165) 17.216 ms 35.136 ms 17.685 ms 7 mny1-cr10.bbnplanet.net (209.244.160.142) 17.600 ms 34.126 ms 17.568 ms 8 p1-0.nycmny1-nbr2.bbnplanet.net (4.24.8.169) 21.034 ms 82.795 ms 115.495 ms 9 p9-0.phlapa1-br1.bbnplanet.net (4.24.10.177) 88.324 ms 45.269 ms 17.581 ms 10 p2-0.iplvin1-br2.bbnplanet.net (4.24.10.182) 35.482 ms 32.461 ms 32.800 ms 11 p9-0.crtntx1-br1.bbnplanet.net (4.24.10.213) 122.929 ms 69.891 ms 90.008 ms 12 p15-0.crtntx1-br2.bbnplanet.net (4.24.10.114) 123.054 ms 215.361 ms 216.327 ms 13 p9-0.lsanca2-br2.bbnplanet.net (4.24.5.61) 90.234 ms 87.366 ms 80.662 ms 14 p1-0.lsanca2-cr1.bbnplanet.net (4.25.111.1) 81.343 ms 124.413 ms 89.594 ms 15 p2-1.lsanca1-aa2.bbnplanet.net (4.25.111.14) 90.254 ms 89.236 ms 90.133 ms 16 4.24.44.34 (4.24.44.34) 81.434 ms 80.709 ms 94.729 ms 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 *^C In this example, it appears that the host after the 16th hop is blocking ICMP echo packets. As the last packet to have valid reverse DNS was a BBN Planet name, you might think that the packets originated with that organization. In fact, that organization (the original name for Genuity) is the ISP providing IP service to the customer at 4.46.94.68. If you have problems with a customer, you may wish to contact the site's network provider. Even if the site's network service provider will tell you nothing, he will often forward messages to the relevant people. In an emergency, you can call the organization's main number and ask the security guard to contact the computer center's support staff. 22.2.6.4 Alternative contact strategiesIf the suggestions provided in the previous sections don't work, consider these strategies for making contact:
22.2.7 Getting Rid of the IntruderKilling your computer's power ”turning it off ”is the quickest way to get an intruder off your computer and prevent him from doing anything else ”including inflicting further damage. Unfortunately, this is a drastic action. Not only does it stop the intruder, but it also interrupts the work of all of your legitimate users. It may also delete evidence you might need in court someday; delete necessary evidence of the break-in, such as running processes; and cause the system to be damaged when you reboot because of Trojaned startup scripts. In addition, the Unix filesystem does not deal with sudden power loss very gracefully: pulling the plug might do significantly more damage than the intruder ever meant to do. You can forcibly kill an intruder's processes with the kill command. Use the ps command to get a list of all of the user's process numbers, change the password of the penetrated account, and finally kill all of the attacker's processes with a single kill command. For example: # ps -aux USER PID %CPU %MEM VSIZE RSIZE TT STAT TIME COMMAND root 1434 20.1 1.4 968K 224K 01 R 0:00 ps aux nasty 147 1.1 1.9 1.02M 304K p3 S 0:07 - (csh) nasty 321 10.0 8.7 104K 104K p3 S 0:09 cat /etc/passwd nasty 339 8.0 3.7 2.05M 456K p3 S 0:09 rogue ... # passwd nasty Changing password for nasty. New password: rogue32 Retype new password: rogue32 # kill -9 147 321 339 You are well-advised to change the password on the account before you kill the processes ” especially if the intruder is logged in as root . If the intruder is a faster typist than you are, you might find yourself forced off before you know it! Also bear in mind that most intruders know how they broke into your system ”they can use that technique again. They may also have installed back doors or other ways of re-entering your system. Thus, even if you change the password, that may not be sufficient to keep them off: you may need to take the system to single-user mode and check the system, first.
As a last resort, you can physically break the connection. If the intruder has dialed in over a telephone line, you can turn off the modem ”or unplug it from the back of the computer. If the intruder is connected through the network, you can unplug the network connector ”although this will also interrupt service for all legitimate users. Once the intruder is off your machine, try to determine the extent of the damage done (if any), and seal the holes that let the intruder in. You should also check for any new holes that the intruder may have created. This is an important reason for creating and maintaining the checklists described in Section 20.3.3. |