Aftermath... Security - A People Problem

Aftermath Security “ A People Problem

Security at Pacific Tech has never been as I, Ben Strobell, would have liked it “ users and systems administrators alike bypassing best security practices in the name of functionality and ease of use. I have always said to my co-workers at the college that security isn t good security unless it sucks. Of course, the less security conscious systems administrators and developers would just laugh at me “ but after the activities on our network over the last few weeks came to light, those guys were left to eat their own words. As I mentioned “ many of the systems administrators here refuse to abide by best security practices in their daily chores. In spite of this, I make every effort to ensure that all of my work conforms to what I believe to be best practices. When I joined the college some twelve months ago, the SIRAM web application (for which I am now responsible) was an utter mess. The TSQL code was just full of user -dependant database queries which the lamest of script kiddies could have exploited in order to read or modify data in the SIRAM database “ heck, the production database was using the database administrator account with a null password! Over and above database- related problems, the application permitted students to upload pictures of themselves to their student profile . Of course, this functionality allowed the upload of any file types, including windows executables, active server pages “ you name it, it was permitted.

So I made it my job to overhaul the entire application “ I wrote a SQL wrapper function, through which all database queries would be passed and checked for the presence of SQL meta- characters , prior to the actual query being executed. Further to this, the application was enabled with extensive auditing capabilities. All user events would be logged; accounts would be locked out for a temporary period after a number of failed logins had occurred; after I had finished that application was probably my best work in years . But of course, information security isn t just about technology; information security is a people problem . And in my opinion it was the shoddy student network and system administrators (such as Frieda Peterman), which this college hires that ultimately lead to the compromise of almost forty three thousand student social security numbers and other miscellaneous student data.

Frieda Peterman and I have never been on particularly good terms. I was hired by her predecessor who also despised Frieda and her shoddy work practices. Of course, shortly after she had been hired , Frieda was immediately promoted to the roll of lead systems administrator “ in other words, my boss. Frieda was one of these people who just love to have control of everything “ If there is a system which she didn t have access, despite whether she actually required it or not, she would kick up a fuss.

One day in late February last year I found myself having to miss a day of work thanks to food poisoning I contracted from a Chinese take away I had eaten the previous evening. Aside from the time I had to have my appendix taken out, that day was probably about the most ill I have ever felt, I just wanted to curl up and die. After I reluctantly received a support call early that morning from my bed I opted to power down my cellular phone and attempt to get some rest. I had no plans to power it back up until I arrived at work the following day.

Naturally, as any systems administrator will have experienced , the day you turn off your pager or cell phone is the day that the network falls apart. Well, the network didn t fall apart, but I was greeted at work by a furious Frieda Peterman who had apparently attempted to call me a number of times on the previous day “ reminding me of my contractual obligations to ensure I am reachable at all times of the day, even if I am on a sick-day. After an hour of attempting to be diplomatic with Frieda, she eventually calmed down and explained that she was just upset because she was unable to upload content to the student intranet server: my.ptech.edu (a server which she has no real business having access to). After coyly inquiring into her reasons for wanting access to the host and immediately having my head bitten off for questioning her, I proceeded to add a user account for her. Of course, a user account was not sufficient for her “ Frieda insisted on using the same mechanism which I use to upload server content. After spending what seemed like an hour explaining the concept of RSA keys and why I couldn t let her use my key, Frieda spent the remainder of the day reading about RSA keys and their use with ssh (secure shell). The following morning Frieda approached my desk with a print-out of a web site she had visited and instructed me to implement the solution to which the web site alluded.

In essence “ Frieda was requesting that I remove the password from my RSA private key to allow her to use the purpose-written script I use to upload SIRAM content. After additional arguing and developing a burning desire to attack Frieda with my newly purchased rubber dart gun from think geek I submitted to her demands in the name of maintaining a healthily low blood pressure. Given my use of the current RSA private/public key pair used by the script for access to other systems, I opted to generate a new key paid, specifically for the use of the SIRAM upload script, removing the password from the private key portion of the pair after generation.

After a quick modification to the upload script to remove the prompt for the private key pass phrase from the command line, I copied the new version of the script to my and Frieda s home directories. Frieda was happy “ so I was happy. Life went on as normal over the following months “ most of my time was being taken up by further developments of the SIRAM application to allow students to sign up for classes online “ a task which would ve previously required a visit to a college office. Both the systems I administer and I breathed a heavy sigh of relief as Frieda was moved to network operations “ leaving me in charge of the systems administration team. Frieda was put in charge of the college project to upgrade all network hubs to layer three switches in support of the new high speed college network backbone which was being put in place over the course of this year.

Following the completion of the SIRAM application update, I decide to take advantage of my new-found position of chief administrator to raise the general level of security on the network through the installation of a distributed collection of network-based intrusion detection devices (NIDS). After having the project approved by the colleges purchasing department, I went ahead and purchased a number of rack mount computer systems. Although I could have purchased a commercially-designed NIDS, the price would have been substantially more “ limiting the number of devices I could purchase. Along with the IDS software which I installed on the stripped down Linux-based system, I also installed a number of third party programs to monitor various network activity. Amongst the programs installed was a small, freely available tool I located named arpwatch . In essence, arpwatch will keep an internal database of all observable ARP activity on the network to which it is connected. If a new MAC address is seen, or the MAC address of a known IP address suddenly changes, a report will be sent via email to a pre-defined address “ in this case, [e-mail], which is an alias to my email account.

The NIDS devices were good to go “ I had tested their performance on a dummy lab network which I had constructed for this purpose in our office. Due to the new layer 3 switches we had recently installed, for the NIDS devices to work correctly, I would have to request that the network administration team set the switch port for the NIDS device to be put in mirror mode. Without this, the only way that the NIDS device was going to see the traffic going through its switch port would have been if I were to flood the switch with ARP traffic, filling its ARP tables “ and I was pretty sure that would not have improved my relationship with Frieda. Accordingly, I put a request in to the network administration department for a spare port on each switch located in each lab to be put into mirror mode and the number of the port emailed to me so I would then know which port I needed to connect my new NIDS devices to.

Naturally, it was Frieda who replied. Ben “ Due to the heavy load our network team is currently under, we will not be able to carry out your request “ if you would like the telnet passwords for the switches so you can carry out this work yourself, please drop me an email and I will have them emailed to you . Frieda s tone was by no means unpleasant “ but was also not the response I was hoping for. Not wanting to have the passwords mailed over to me, I declined Frieda s offer and informed her that I would get the passwords from her on my next visit to their office.

Downhearted that I was now unable to install my new NIDS devices due to a lousy network administration department, I decided to spend some time re-auditing the SIRAM application, for which I was wholly responsible. I spent a number of minutes reflecting on the changes I had made over the past few weeks. I had installed a copy of web-cvs on a local web server. This allowed me to easily view all code changes that had been made. As I was now the sole developer on the project, I had configured a cron job which would CVS update the files in my development directory on an hourly basis. This would ensure that I had an audit trail of any ad-hoc changes that I made to the code and had forgotten to back up. If I made a mistake and broke something, I could always fall back to the previous “ known- working copy.

On browsing through the most recent changes, I noticed an anomaly in an area of the session tracking code “ a part of the application which I had not changed in at least two months. Curious about the nature of the changes which I had supposedly made, I used web-cvs to check on the differences between the two code versions “ this would have had a similar effect to downloading both file versions and executing the diff command.

 *** 1,6 **** ! if($s_cookie == "404280206xc492734fa653ee9077466754994704fL") { !       $cmd = "SELECT SSN, FIRST_NAME, LAST_NAME, STREET, CITY, STATE, ZIP,  PHONE, EMERCONTACT_NAME, !               EMERCONTACT_PHONE, EMER_CONTACT_STREET, EMER_CONTACT_CITY, EMER_CONTACT_STATE, EMER_CONTACT_ZIP from USERS"; ! } else if($s_cookie) {         $cmd = "SELECT SSN, FIRST_NAME, LAST_NAME, STREET, CITY, STATE, ZIP, PHONE, EMERCONTACT_NAME,                 EMERCONTACT_PHONE, EMER_CONTACT_STREET, EMER_CONTACT_CITY, EMER_CONTACT_STATE, EMER_CONTACT_ZIP from USERS WHERE id='$s_uid'"; --- 1,3 ---- ! if($s_cookie) {         $cmd = "SELECT SSN, FIRST_NAME, LAST_NAME, STREET, CITY, STATE, ZIP, PHONE, EMERCONTACT_NAME,                 EMERCONTACT_PHONE, EMER_CONTACT_STREET, EMER_CONTACT_CITY, EMER_CONTACT_STATE, EMER_CONTACT_ZIP from USERS WHERE id='$s_uid'"; 

To my surprise, over the last two days, the code had apparently been changed on two separate occasions “ to add and remove the highly questionable code. Although it was clear what the code did, I uploaded the changed version of the code to a development web server and, sure enough, when a request was made with the cookie value of 404280206xc492734fa653 ee9077466754994704fL , all row sets for the student information (USERS) table was sent to the web browser. I immediately disconnected the network cable from system which the apparent code change had occurred on and begun to search for signs that a compromise of that host had occurred.

After spending a number of weeks investigating what had gone on with a now overly-cooperative Frieda Peterman who was keen to do all that she could to ensure that she was not relieved from her position as the network administrator, an examination of the log files on various hosts and the audit logs from the SIRAM application revealed that an individual, presumably a student had been accessing the SIRAM accounts of multiple students from an IP address which was, at the time of the investigation, not bound to any known system on the college network.

It was apparent from the log files that the SIRAM account compromises had occurred prior to the shell account compromises on various UNIX systems around the college network. After postulating toward several possibilities regarding how the SIRAM account may have been compromised, a timid Frieda admitted to not having enabled port security on the new switches which she had overseen the installation of. Port security ensures that only a pre-configured MAC address may talk to a respective port on the switch. Although real supporting evidence was lacking, it now seemed that the most likely possibility was that a student had some how connected a rogue system to the college network and potentially hijacked one or more gateway addresses via ARP poisoning “ enabling a multitude of attacks to be leveraged in order to steal the SIRAM web application authentication credentials.

In addition to the obvious breach of security within the SIRAM application, my investigation also drew me to the number of student registrations which had been occurring from a single IP, allocated to one of the universities labs. When I say a number of, several hundred registrations appeared to originate from that IP in just a few days. The university had not made use of any kind of inline proxies and there were no networks within the university campus configured to use any kind of NAT. The only logical explanation was that someone had installed some kind of proxy “ hm perhaps that was how the attacker retrieved those user accounts. To add insult to injury , after describing my theory to Frieda, she responded with a shy admission that until shortly before the SIRAM compromise, the SIRAM application had been operating over SSL using a self signed SSL certificate. This came as no surprise to me, Frieda was clearly not cut out for systems administration “ but she learned a valuable lesson, despite the cost to our college. So the investigation drew to a close and a lack of evidence prevented the attack being traced to a student at the college.

Since then, I have developed an interest in a number of methodologies which I had previously read about regarding developing threat models for computer networks and ways in which attackers can be characterized during post-incident investigations. As the SIRAM incident proved, the previously unrealized threat which the college had in the past neglected to mitigate against, was the insider “ our own students.

Subsequently, the security posture of our college has changed substantially. As the newly appointed head of security, I am now in a position to ensure that (as well-intentioned as they may have been) people like Frieda Peterman are no longer able to do things such as authorize the use of passwordless RSA keys for access to critical systems and, more to the point, people like Frieda Peterman now understand why .

As for me “ I am still curious to why exactly it was that a student was compelled to retrieve the personal records of our students. The adversary with whom we are dealing is clearly reasonably skilled or well resourced. His preference to risk is such that he was sufficiently motivated to retrieve the student data and that he was oblivious to the fact that he might have been expelled from the college if caught. I fear that a far more dangerous being than a student may be at work amongst our community.

* The Author would like to acknowledge and thank Neal Israel, Peter Torokvei, and Dave Marvit, for the wonderful movie Real Genius , without which this homage would not be possible.