Partner Connections

JFC Pharmaceutical wanted to share data with one of its customers to facilitate a joint research project. McConnell's Drugs needed access to the information stored on a database server (Drug10). The technical end, providing the necessary connection to the customer, fell into the hands of the system administrator, Dave Furlong.

Since Dave had never worked on a project of similar scope, he began by looking for documentation. He found that JFC did not have an approved architecture or policy for connecting customers to their intranet. So, Dave asked for advice from the company's firewall expert, Fred Johnson. Together, Fred and Dave came up with their own plan. They connected the database server to the Internet so that McConnell Drug's employees could access the data. Unfortunately, they connected the database server directly to the Internet without a firewall in front to protect it and without security configured on the database server. That configuration left the door to JFC's network wide open. Clearly, it would only be a matter of time before a hacker walked right in. And that's exactly what happened a hacker walked right in.

How could a firewall administrator and a system administrator make such a serious mistake? Why did they even have the power to make those types of decisions?

Frighteningly, this type of thing can happen. When companies lose control of their external connections, and network lines become blurred, one mistake can destroy the future of an entire company. And that is what nearly happened to JFC.

Day 1: Security Architecture

Fred and Dave met to decide on a workable access approach. Since Fred had nothing written on how to connect customers to JFC's network, they discussed sharing the data on Drug10 and the possible approaches to solving the problem. Basically, Fred and Dave decided to connect (database server) Drug10 directly to the Internet so that McConnell Drugs could access it from there. They figured that they would configure Drug10 to serve two purposes. First, Drug10 would serve as a firewall. Second, and most important to Dave's needs, McConnell Drugs would have access to the information it needed.

Since Fred had more experience installing firewalls and connecting systems to the Internet, he agreed to help Dave.

A Few Weeks Later: Security Installation Policy

Dave finished his part of the process first. Drug10 already had poor response time. So, before connecting it to the Internet, Dave installed a powerful new system and loaded the software. Since Dave didn't have any policies or procedures for connecting systems to the Internet, he just did an out-of-the-box install. In truth, Dave had no idea how to configure security on the systems, so he figured that fell into the hands of the firewall expert, Fred. Unfortunately, Fred didn't know that.

The Next Day: Who's Responsible for Security

Fred connected the database server to the Internet. Then, he gave McConnell Drugs access so that they could copy files from Drug10 to a system on their network. Fred never bothered to configure security on the system, because he assumed it was Dave's job. As far as Fred was concerned, once everyone had access to what they wanted, his job was done. Fred moved on to another project.

Over the Next 29 Days: A Hacker Gains Control

It was only a matter of time before a hacker discovered the unsecured server. The hacker broke into Drug10 and gained full control of the database server. The hacker replaced critical system files and left back doors on the system for easy access at the next visit.

At this point, McConnell Drug's network was also at risk. They were pulling down information from a system that could be infected with a virus, worm, Trojan horse, or the like. Even assuming that the hacker was not malicious (a pretty risky assumption), the results could have been devastating. Imagine coming in Monday morning to find that all your company's human resources data had been posted to the Internet. Imagine the profound embarrassment of employees who had expected their personal data, salaries, and job performance reports to remain confidential. Now, remember where we live. As nearly everyone knows at this point, embarrassed Americans don't get even they get lawyers!

Speaking of lawyers, the hacker that wandered through JFC's intranet could have destroyed all of JFC's data and infected or destroyed McConnell Drug's data, too. Just think about the liability there. The ultimate responsibility for destroying data obviously rests with the hacker. But, corporate lawsuits are often based on who can pay as much as who should pay. No doubt, JFC's financial backing would provide a better, and bigger, legal target than the hacker's assuming that the hacker could be caught in the first place.

+ One Month: An Unscheduled Security Test

Drug10 remained connected to the Internet for a full month before anyone discovered that it was compromised by an intruder. The discovery was made by pure luck. I made it when I was hired to conduct a routine audit on some of JFC's systems. Without that good timing, the compromised server could have gone unnoticed and unprotected indefinitely.

My involvement began when JFC's management hired me to test security on some of the servers in their computer room. Although the compromised system (Drug10) was listed as a database server, it was not one of the systems that I was hired to test.

JFC had hired me for what insiders call a "spot audit." Some companies use a spot audit to get a handle on their level of risk. In a spot audit, you basically select a representative group of mission-critical systems. If the audit shows that the systems are at risk, chances are pretty good that the rest of the systems are, too. It's a cost-effective approach to security testing not in the same league as a complete audit, but certainly better than relying on luck (the current security strategy used by more companies than you'd realize).

The problem with a spot audit is that you could be looking at the one clean spot in the room. You have to be careful. When I'm hired for a spot audit, I try to keep my eyes open to the room around the spot.

Audit Day 1: Network Maps Tell a Lot

I had asked management to provide me with a network map. The map was waiting for me when I arrived. I like to see more than a list of systems and network numbers before I begin an audit. I like to see where all of the connections go. To do that, I need a current network diagram something that shapes the virtual world into a more tangible form. I also consider network diagrams to be a fundamental element for maintaining connections to a network. If the system administrator tells me that they don't have a map, or that the map is in her head, I get worried.

JFC had a nice network map. After glancing at it, I noticed that one of the database servers connected to JFC's intranet was also connected to some other unlabeled network. Where did the other network go? Obviously it went somewhere, but from the diagram I couldn't be sure where. The network line connected to the system pointed off into the air, in the same direction as the company's firewall. By the way the map looked, the database server seemed to be connected directly to the Internet.

Connecting a database server to the Internet without security configured or a firewall in front of it was a ridiculous thought. No one would ever do that! Or would they?

When I conduct an audit, I've learned not to assume anything. It's best not to make assumptions about security. In fact, an assumption is what caused the problem in this case. Doug assumed that Fred would set up the database server as a firewall. Fred assumed that his job was just to connect it to the Internet. Having discovered that, my job was to contact management and let them know that I needed to expand the scope of my audit to include Drug10.

After reviewing the network map and targeting the systems that appeared to be high risk, I met with Dave to find out which systems he thought I should audit. It's important to find out what the people in the trenches have to say. They may know about some of the hidden risks the company's dealing with. Dave said it didn't really matter to him which systems I checked.

Some system administrators don't like their systems to be tested by auditors. People sometimes think they are going to lose their jobs, but that's not what security auditing is all about. It's about reducing risk, improving security, exercising due diligence, and so on. I wanted to reassure him that I would let him know what the problems were, if I found any. I also reminded him that sometimes a security audit can improve security as well as show the need for increased funding and head count. I asked Dave whether he had enough help configuring security. He said that he really didn't know how to secure the systems and that he needed to get help from other members of the support staff.

I told Dave that it sounded like he could use a little security training. He thought that idea was great. Like most system administrators, Dave rarely had time to take training, because he spent most of his time just keeping the systems up and running. In short, Dave was untrained and understaffed. That was no surprise to me. System administrators are supposed to know everything and work all of the time. I know, because I used to be one.

I told Dave that I had a network diagram, but that I still needed their security policies and procedures. He said he'd get me copies by the next morning.

I still needed to ask Dave about the database server. He was already a little anxious, though, and I didn't want to send any bad signals. So, on the way out the door, I casually added, "By the way, the database server called Drug10 looks like it's connected to two networks. Is it?" Dave said, "Yes, I connected that server to the Internet so that one of our customers could access the database." That further piqued my curiosity. "When did you connect the system?" I asked. Dave replied, "Last month."

Hmmmm... "I've never seen anyone connect a database server to the Internet without firewall protection." Dave responded, "Our firewall expert, Fred, configured Drug10 to act as a firewall after I installed the system."

Given that, I found it hard to believe that these men hooked their system up to the Internet. Talk about risk! Since I had enough information to start testing, I decided to interview the firewall administrator, Fred, after I tested the security of the system. I wanted to have all the facts before I met with him. And Drug10 was calling my name. I had to go.

It was already 5:00 p.m., so I was going to have to wait until the morning to test out the systems. Dave had to pick up his child and wasn't willing to hang around the computer room so I could audit their systems. However, he did agree to set me up an account on Drug10 first thing in the morning. I could be ready to pounce as soon as I arrived. Arrangements made, we left at the same time.

Unlike Dave, who seemed happy to leave, I was unhappy about the wait. I had to clear my mind, or I would go crazy waiting until the morning. A good run would do exactly that. I was home lacing my running shoes before I knew it. I bolted out my front door and went for a hard run all hills. After five miles of running the Portola Valley hills, nothing else matters. It was a strong, healthy run. And it worked! The next morning came around fast.

Audit Day 2: Unenforced Policies

Dave met me in JFC's lobby. I wanted to get my hands onto Drug10's keyboard, but first I had to obtain the security policies. We stopped by Dave's office to pick them up. He had printed out quite a few policies for me a few inches' worth. Dave sat down and started reading an e-mail message. He started swearing a bit and pounding away a reply on the keyboard. He must have been dealing with a tough customer.

While he did that, I had a chance to peek at the policies. They were high-level policies the 30,000-foot kind that help executives draw a line in the sand. Enough of a policy for the CIO to feel good, but not enough to support or protect the company at the line level. What I saw also didn't constitute a firewall policy it was more of a connection policy. Basically, the policy mandated only one firewall connection to the Internet and the company's intranet. No exceptions. So much for following policy.

As soon as Dave took a break from his e-mail, I asked him to let me into the computer room. JFC had several levels of security protecting the computer room. They also had four computer operators watching consoles and other equipment. They had me sign in, and Dave verified that I would be working there for the day. Physical security was good. These guys had invested some serious bucks in it.

Dave and I walked down the middle of the computer room. There were servers towering over my head in all directions. They all had names posted on them Drug4, Drug5, Drug6. I could feel the adrenaline starting to race. I knew what I was going to find. I can smell risk a mile away. There it was I could see Drug10 around the corner.

Dave gave me my account and said he'd come back to let me out for lunch. I was already logged in by the time he finished his sentence. As I probed the system, I could hardly believe my eyes. It was true. This system was connected to the Internet without any security configured. It was another out-of-the-box install.

Further examination showed that the server had been severely compromised by a hacker. The hacker had even replaced system files and left back doors to facilitate an easy return trip! It was impossible to tell if the hacker came from the Internet or JFC's intranet. Since root access was easily attained on the Drug10 server, the hacker didn't even have to work very hard. The hacker's job was also simplified by dormant accounts with easily guessed passwords and the fact that security patches had never been applied.

The Drug10 server was also running network services that should have been disabled. There are many ways to gain information about a system by exploiting network services. For example, the "finger" command provides information about system users. That information could later be used to launch an attack against those users. Why provide information if you don't need to? That's one reason why you should disable the services you don't need to run.

The hacker's job was not just simplified: it was a walk in the park. There were so many ways to break into this system, I couldn't begin to guess which method the hacker used. Maybe a different one each day for variety? Something had to be done right away.

Since the risk to JFC (and McConnell Drugs) was so high, there wasn't time for a written report. Some auditors color-code risks as green, yellow, or red. I had a category for this security risk called "THE SKY IS FALLING NOW."

I got in touch with Doris, the internal audit manager who had hired me, and told her the sky was falling and why. She contacted the key players and called an immediate mandatory meeting. The attendees were Dan (JFC's security expert), Fred (JFC's so-called firewall expert), Dave (the system administrator), the managers of all support groups involved, and me. Within two hours, all the players were assembled in the same room.

One of the things I like about dealing with professional internal auditors is that they understand risk and they have enough power inside a company to pull the plug from any system if they need to. My recommendation was to pull the plug on Drug10 immediately. Thankfully, the internal audit manager understood the severity of the problem. The support staff worked the entire night installing a new system to replace Drug10. By the next morning, that new system was online.

The Last Audit Day: Taking Responsibility for Security

With the risk to JFC's network reduced, I was able to complete the rest of my audit. My audit results showed serious security violations on most mission-critical systems.

At the top of the risk list were:

• Security patches were not installed.

• Excessive file permissions existed.

• Passwords were easy to guess.

• Unnecessary network services were enabled.

These risks didn't surprise me. When an important system like Drug10 is misconfigured to the point where it places the entire company and its customers' data at risk, I don't really expect to find adequate security controls on the other systems.

Before writing my report, I decided to stop by and talk to Fred, JFC's so-called firewall expert. Stopping by his office, I asked for a few minutes of his time. As I sat down and began to make small talk to open the conversation, Fred kept typing. I really hate that. When people type over your conversation, they're announcing a couple of things: (1) I've got more important things to do than talk to you; and (2) you don't really require (or deserve) all of my attention. I wasn't in a mood to be acknowledged half-heartedly, so I got up and asked Fred to meet me in the conference room at the end of the hall.

Fred followed at his own good pace. In probing Fred for information, I found him to be sarcastic, snippy, and not very intelligent. In my opinion, he was a real loser. He also made a point of passing the buck. He let me know in no uncertain terms that everything would have been fine if Dave had configured the security on the database server. He said it was never his intent to configure the security of the system that was Dave's job.

If you happen to be a system administrator like Dave, that's something you should probably remember. When security breaks, the finger almost always points back to you.

I didn't want to waste too much time talking to Fred, since I still had a report to write. However, I did decide to ask him a few more questions just because I knew it would irritate him. (Okay, not very nice on my part, but we should all get to have a little fun from time to time.)

It turned out that Fred had been working at JFC, maintaining the network and configuring firewalls, for five years. For JFC, that spelled five years of major security risks. Ernst & Young reported in 1996 that over 20 percent of surveyed companies have no employees dedicated to security. What they didn't report is that having the wrong person can be just as bad or worse. With no security experts, at least users aren't deluded into believing that their data is safe when it isn't.

Hiring the wrong person for a security job can put an entire company at risk, especially if management doesn't know enough about security to know when to hold their security expert accountable for lapses. Fred's manager seemed especially clueless. I don't think he even understood risk.

And the risk at JFC was substantial. Because of poor configuration, you couldn't even tell whether the hacker had stolen proprietary information to sell to a competitor. Likewise, you couldn't tell whether he'd left behind a Trojan horse, worm, or virus that might later infect JFC's data and that of its customer, McConnell.

As for external connections, it was impossible to tell where the network started and ended. If you worked for JFC and wanted an external connection, all you had to do was call Fred. Fred personally authorized all connections and stored information about those connections in a flat file. You couldn't run any reports on connections, and it was hard to search for anything in the file. Overall, it was just one big mess.

It took me a few days to complete an executive-level summary for JFC. It took a while to write, because I needed to condense a lot of risk factors into just a couple of pages. Those risks included inadequate approval processes for external connections, an unenforced firewall policy, poorly developed policies and procedures, and an overly risky system configuration. The report also addressed poor training, ineffective management, and failure to track external connections. I handed the report to Doris and left. Now it was her responsibility to make sure those problems were fixed.

Summary: Keep the Competition Out

In this case, the problem was found before anyone got hurt. Of course, not every company is lucky enough to find these types of problems during routine security audits of other systems. Many companies never bother with security audits routine or otherwise.

Employees like Fred and Dave really do exist. Not everyone has the best interests of the company in mind. That's why basic policies, procedures, and controls are necessary to protect companies from simple mistakes that could literally destroy them. Without those factors, the big picture gets blurred.

As the JFC case shows, that blurring can happen very quickly. JFC was a company on the rise. Their new drug formula was poised to blow the doors off their competition. What if the hacker who broke into Drug10 had been an industrial spy working for a competitor? Or a foreign government hoping to give homegrown companies an edge? According to Michael Anderson, a former treasury agent who now sits on the advisory board of the National White Collar Crime Center, even friendly nations aren't so friendly when it comes to industrial spies. According to Anderson, the French have been known to bug both the first-class sections on airplanes and the luxury-hotel suites favored by American corporate executives. Other heavy players in the game of industrial espionage include Japan, Israel, and a slew of former KGB agents.

A proper firewall, secure system configuration, and detection mechanisms (security components missing from Drug10), are a first line of defense in closing the door against industrial espionage, competitors, and other adversaries.