Overlooking Training

InterMint Financial was a well-known Wall Street giant with over a thousand systems on its intranet. Because of its size, support responsibilities for that meganetwork were split between five different people: Jose Garcia, Dawn Forbes, Kenji Abe, Smita Kumar, and Tia Fairchild. Unfortunately, only Jose had been trained in how to configure network security.

Jose received good security training when he came on board. After getting that training and some hands-on experience, Jose knew what it took to run a tight security ship. He configured his systems with security in mind, installed auditing, used intrusion-detection software, monitored security aliases, and generally kept an eye out for new security alerts. Jose took an active approach to securing his network.

If everyone had known what Jose knew, InterMint's intranet would have been in great shape. However, it wasn't Jose's job to teach the rest of the staff how to support security.

Like many companies, InterMint didn't have a security training program for system administrators. Oh, the training looked good from a high level. They had a great information-protection training program for other employees. They even sent out training information on selecting good passwords. But they didn't offer even a single class in security for the people installing the systems.

The only reason that Jose was properly trained was pure luck. When Jose first started, he found out that someone had broken into the CEO's system. In response, management dished out the funding and training for Jose so that he could keep the executive network secure. The other four system administrators, who were supporting the legal, treasury, security, and trading-floor systems, didn't get any training.

Since the other four system administrators had no idea how to configure or support security they didn't. That left the information on their networks open and available to anyone! Of particular concern was that the e-mail and home directories of all the systems, except the executives', were left open for anyone to read, copy, modify, or destroy. Unfortunately, that's what happens when you leave corporate networks in the hands of inexperienced system administrators. What you don't know can hurt you!

Of all the tasks faced by today's system administrators, training is probably the task for which they're allotted the least time. Overworked and understaffed, many system administrators see training time as downtime stolen away from the more urgent tasks of routine maintenance and general care and feeding of the network. Of course, in many respects, training is an essential type of routine maintenance for systems workers. When that maintenance is postponed (or just ignored altogether), the results can be surprisingly dire.

Initial Contact: Security Testing

A few years ago, the internal audit department of InterMint Financials hired me to test security at their corporate office. Being hired as a contractor by a company's internal audit department is not unusual. Internal audit departments usually don't have a security expert on staff. Instead, they contract security work out to experts who report the risks back to them. This particular department asked me to take a look at the corporate network and to report back any risks that I found.

Unlike most of the audits that I do, this one wasn't prompted by a known problem. Company officials just wanted to make sure that the level of risk on the corporate network was low. At first, this audit seemed like a breath of fresh air. A company that wanted an audit, not because it was reacting to a known problem, but just to make sure that there weren't any unknown problems!

Day 1: Gathering Facts

I began this audit, like most, by going over the network map to get a feel for configuration and possible risks. I like auditing corporate networks because that's one of the places a hacker will look for corporate secrets. These guys pay me to dig up dirt. The best way to do that is to think like a hacker. Since I wasn't auditing an engineering network, I didn't look for source code or advanced R&D. Instead, I looked for high-level strategic information. Basically, I did what a competitor pretending to be an employee might do. I looked for easy access to executive systems, searching for corporate strategy information or even personal information about the executives. Because competitors can and do take this approach, CEO, CIO, and CFO systems should always be considered potential targets. To keep hackers out of these areas, extra precautions must be taken during installation and routine auditing to reduce the risk.

The first time I spoke with the internal audit manager, Randall Millen, I was impressed that he wanted to conduct a security audit just to make sure that risk didn't exist. My audit, however, marked the first time ever that the internal audit department had audited the security of the corporate network, so I fully expected to find risk. That is not to say that the company struck me as particularly lax on security issues. Companies that don't audit the security of their networks have no proof of network security. They're just assuming that the networks are secure. In my experience, that assumption itself is a major security risk.

I continued to examine the network layout and the type of data being stored. It yielded some pretty juicy stuff. For starters, all the executive data was stored on the network. The bonus points, however, were the legal, treasury, and corporate security systems, which represented a tempting haul of proprietary data. My plan was to systematically target each system in these groups, focusing on the tastiest proprietary data. Given the nature of the data, any success on these systems would surely raise an eyebrow or two.

An interesting trait of the network layout was that each department's data was stored on a different network supported by a different person. Executive systems were controlled by Jose Garcia, legal systems by Dawn Forbes, treasury systems by Kenji Abe, the trading floor by Smita Kumar, and corporate security by Tia Fairchild. That support structure alone added risk to the picture. Among other factors, the possibility exists of poor training or support procedures for each system administrator in charge of a network. I know it's impossible for one system administrator to support thousands of machines accurately, so companies usually have more than one support person in the picture. But I also know that the more people there are in the picture, the higher the risk generally is, and, of course, the more opportunity for some enterprising hacker. Make no mistake, hackers know about the fallout risk from multiple support personnel. That's one of the things hackers dream about tons of system administrators who know little about security supporting mission-critical machines.

Day 2: Testing the Systems

After completing my initial research, I decided that this audit would require a penetration test. I had three reasons for including that test. First, the audit manager had no idea how much (if any) risk existed on the network. That told me that the executives were probably just as clueless on the state of security. They probably thought it was OK because no one had told them differently. Clearly, this was one of those times that I needed proof of concept to show to management. I needed to prove the concept that someone could break into their systems. Second, I admit that I thought these would be fun systems to break into, just because of the nature of the information stored. My last reason was that I had some new toys I wanted to play with. Brad Powell, a known force in security circles for years, had just passed me some great new break-in tools.

Like hackers, security professionals often pass each other new tools for breaking into systems. Of course, you have to be in the security loop to get them. That's part of the security professional's code of ethics. In any case, I had already tested the tools on my network and they worked like a charm. Now I had the chance to try them out in the "real world."

I began my audit by probing for information on the executive systems, then ran through my usual tests for holes. (Basically, I did what a hacker would do.) Surprisingly, I found that the executive systems were pretty solid. Jose obviously knew his stuff and took the time to make sure that his systems were secured. Without a doubt, he had been trained well and knew exactly what it took to support his executives' machines. Of course, some security experts would argue the point, boasting, "I can get into any machine." During my penetration test, however, I exploit all the well-known vulnerabilities. If I can't get into a system after my routine, the system passes that test. And Jose's systems passed my tests with flying colors.

Continuing on with the executive systems, I asked Jose for a login account. I also let him know that he did a good job. System administrators rarely hear those words from a security auditor and Jose's smile let me know that he appreciated them.

I logged into one of Jose's systems and looked for basic filesystem and configuration errors. These are the kinds of errors that can't be detected from the network, like excessive file permissions, dormant accounts, and setuid (set user ID) programs. Overall, the system looked good. There was some room for improvement, but nothing that I would note in the security report. I talked to Jose offline to let him know that he could tighten up the permissions on the filesystem for most of his systems. Otherwise, he'd done a brilliant job. And that's the message I planned to deliver to management: "This guy's a star!"

Jose's colleagues didn't assess nearly so well. The legal systems were embarrassingly vulnerable. It took me only a few minutes to gain full control of the first system. Within 15 minutes, I was leisurely meandering through all the corporate attorneys' systems. Dawn obviously had a much different approach to installing and maintaining systems. It almost seemed as if Dawn (or her manager) didn't feel that the legal systems were important enough to secure or monitor. No doubt, InterMint's lawyers would have been appalled.

I found all the legal systems to be wide open. Either Dawn didn't have the time or desire to configure security, or she didn't know how. Auditing and monitoring were also missing in action no one spotted me hopping from system to system looking at all their legal secrets. I'm sure the competition would have loved a peek at some of the data I wandered past. I captured enough evidence (access to restricted files and legal documents) to give the legal department nightmares and then moved on to the treasury systems.

Kenji's systems were no better protected than Dawn's. Within just a few minutes, I had access to the first system in the treasury network. Within minutes, I had control of all treasury systems. Did the company consider legal data and financial data unimportant to secure? Or were Kenji and Dawn simply clueless? My guess was that management didn't understand security and the risk to their data, and never took a serious look at the security controls on the data on their network. If they had understood that all of this information was accessible to anyone on the network, I'm sure they would have flipped out, especially when you think about who, besides full-time employees, is really on your internal network. Contractors, customers, consultants, temps, and outsourcing staff are obvious choices. Depending on the type of business and corporate culture, the list can be longer.

The state of Smita's systems was even scarier. I was able to gain full control and access information on her systems, just as on the last two networks I tested. But what was really frightening was that I could have also changed the password stored in the hardware PROM and shut the systems down. Smita would have been locked out of her own systems until either I gave her the password, or she physically replaced the hardware PROMs.

With a little creativity (and a lot fewer ethics), a hacker could have held the entire trading floor hostage until they transferred a few million dollars to an overseas account. It might have cost them several billion dollars to be down for a week, so what would a few million mean to them under those circumstances?

At this point, I was starting to worry, because breaking into systems at this firm was really much too easy. Surely the last group, corporate security systems, would be a challenge. After all, these systems should have controls on them. Right?

Wrong! Tia must have learned to "secure" her network from Dawn, Kenji, or Smita! Once again I gained full control of the systems without being spotted. And some of the data I accessed was juicy indeed. Among other items, I was able to access data files detailing ongoing investigations. Just imagine being a crook under investigation by the company. With just a little bit of security knowledge, you could get the inside scoop on any investigations on yourself. Nip a little data here, change a little data there, and voila no more security issues for you! Investigation closed.

By now, I had enough evidence to complete the testing portion of the audit. I had also had my fill of reading sensitive data for the day. Like most security professionals, I really care about security on networks. Perhaps too much. A large network like this with such an extreme level of risk was really depressing me. Since the corporate network had been installed for five years, I believed that this risk had most likely been present the whole time.

A lot of people think that security auditors enjoy finding sloppy security and risky systems like it gives us a reason for being. In truth, it sometimes makes me crazy to be able to break into one system right after the other. In any case, I had more than enough "reason for being" for one day.

I packed up my stuff and headed for a local sushi bar (Higashi West). The sushi chefs, Richard, Craig, and Garth, always greet me with a smile at the end of a dark day. As a bonus, they make the best sushi in Palo Alto. I knew that a little sushi, some sake, and a stimulating conversation would allow me to forget about the depressing security on corporate networks. And it did.

Day 3: Leaving Security Training out of the Budget

Before I knew it, my alarm clock went off. It was 4:30 a.m. just enough time for a good run before work. I wandered out to the gym in a coma. In the back of my mind, though, I was already going over the day's plan to complete the audit for InterMint.

After my workout, I headed to InterMint and met the audit manager, Randall Millen, in the lobby. He signed me in for the day and set up times for me to interview the system administrators.

I didn't say much to Randall about my findings. I simply told him that I had some juicy stuff to report later. I didn't want to say too much too early. Randall gave me one of those looks that let me know he thought I was doing a good job. Little did he know that my work would force him to acknowledge risk, increase funding, and most of all provide training.

Early on, my gut had told me that this was probably a training problem. I know what it's like to work in the trenches as a system administrator. In many companies, training is a luxury. In addition to being fast, smart, and able to handle abuse, system administrators are also expected to be all-knowing by osmosis or good karma or any other method except formal training. Administrators who don't fit that bill are eaten alive kind of like sushi.

I met with each of the system administrators (separately) to find out why, in their opinions, the executive network was secure and the rest of the systems were left wide open. Always looking for expert input, I started with the system administrator who supported the executive systems. Jose obviously knew what he was doing and I wanted to know how he obtained that information when no one else in the company did. When I interviewed Jose, I found him to be very sharp. He was warm, friendly, and easy to talk to. He also knew just how important the executive systems were. Apparently, the CEO's system had been compromised by an intruder at about the time Jose started. He quickly learned just where and how security was missing. At that time, the company added security controls and provided Jose with full training on maintaining the security of the network. I asked him if he had any idea how the rest of corporate security looked. He said, "No," but added that he doubted that security was very good, since none of the other system administrators had been trained in security. He doubted (accurately) that any of his colleagues knew how to make their systems secure.

Interviewing the other system administrators confirmed Jose's suspicion. Not one had been trained in security. Tia hadn't even gotten the basic system administration class until six months after she'd been hired! That's a big problem for someone like Tia, because she had never supported the UNIX platform before. Her success in configuring security on the systems depended on how well she was trained by one of the other system administrators. Since the other system administrators didn't know how to configure security, Tia hadn't learned anything from them.

The one thing that Dawn, Kenji, Smita, and Tia had in common with Jose was that they were all very smart people. All had programming experience; Tia and Kenji even had computer science degrees. We weren't dealing with uneducated people. They were simply tossed into their support positions, like so many system administrators, and told to learn it on the job.

This "sink-or-swim" approach to security seldom works. You can't learn how to secure systems on the job if no one is currently doing it. The practice needs to exist and be well established. At the very least, someone who knows, like Jose, needs to teach the rest of the crew the how's and why's. That is, how to configure basic security and why a higher level of security needs to exist on some systems. Ideally, the rest of the crew should also take a professional security class to learn exactly how to configure and support networks without compromising security.

The real blame here fell at two levels. First, management should have provided the necessary training. As much as managers should think about security, however, they often don't. And, they almost never read minds. Dawn, Kenji, Smita, and Tia should have pointed out the security gap and asked for the training they needed. They never did that.

I had now gathered enough information for this audit. I spent a few days writing the audit report. Sometimes it takes just as long to write the report as it does to do the actual work. It's important to point out the risks and recommended solutions in ways that management can understand and that empower them to take action. That's why it takes time. I have seen some auditors use a standard template for each audit, filling in standard boilerplate with standard words for their findings and recommendations. Those people should find a new career. Companies are not paying for a standard template of problems and solutions. They want this stuff customized so that they understand the risks in their environment and how to implement the solutions in their environment.

After writing a custom audit report, I turned my work over to the internal audit manager who hired me. Randall was pleased with my work. He was not pleased with my findings, but he understood the risks and knew what had to be done to fix the problems. Report in hand, Randall headed off to meet with senior management. My job was done.

Summary: Make Sure You Fund Training

Training is the glue that holds security programs in place. For its price, it's pretty cheap glue. Unfortunately, managers often forget to compare the cost of security training to the cost of cleaning up after a major incident. A British study conducted by Price Waterhouse Coopers found that 44 percent of the firms surveyed had suffered attacks in 2001, with cleanup costs averaging $30,000. By contrast, 70 percent of those same firms spent less than one percent of their budgets on security. American firms fare only slightly better in security training. Expenditures are hardly a drop in the bucket to combat a computer crime wave that was estimated in 1999 as already a $10-billion-a-year business.

Computer crime and systems security attacks have reached a level at which the economic health and well-being of the nation are put at risk.

If your company does not have adequate training programs in place, you may yourself be part of that national risk. At the very least, neglecting to provide formal security training to system administrators can put your company at risk. There must be clear guidelines on how to train system administrators on system configurations, policies, and procedures. Most important, someone must be responsible for ensuring that all staff actually complete the training program.

An alternative is to send employees to external training classes. If your company adopts this route, make sure that the budget for training is adequate. Too often, the training budget is one of the first to feel the ax when cuts are needed. But those cuts can come with heavy risks. Saving money on training may make your numbers look good this quarter. But costs can skyrocket if the data on your network is destroyed.

Above all, remember how important training is. Scott Ramsey, Ernst & Young's national director of information security services, notes, "Organizations bring in technology so they need fewer people. But management often doesn't take the time or spend the money to train people in how to use or protect the technology." System administrators aren't born with security awareness. Most need to be taught how to configure security.

Also remember that training is an ongoing process. Just as it's hard to learn everything on the job, it's hard to learn everything in a one-week class. Make ongoing training a part of standard "employee" maintenance.

Last, be sure to leave a clear chain of command. Obviously, you don't want to skimp on the number of support personnel. But when multiple administrators are required for corporate networks, make sure that security and policy decisions are run through a central committee or system administrator. If each administrator sets policies and procedures independently, the risk of error increases with the size of the staff.