Responsible for Security

When Global Chips connected to the Internet five years ago, they installed a firewall. That firewall served its purpose at the time it gave employees access to the outside world and it kept hackers out. Technology changes fast, however, and Global Chips didn't maintain and upgrade the firewall properly over the years. That left the door to their network open.

One day, a hacker walked past that firewall as if it didn't even exist. Then the hacker wandered freely about the company's intranet, collecting passwords and data. The support staff tracked the hacker through the network, but were unable to obtain enough information to trace back to an individual. The firewall administrator, Joseph Withers, did, however, figure out how the hacker breached the firewall and fixed the problem.

Unfortunately, the firewall saga continued. Global Chips was faced with a series of break-ins. After a while, the firewall became a routine target for hacker attacks. After each break-in, Joseph fixed the newly discovered problem. But all attempts to track the hacker back to a real person were useless (which is not unusual). So, Joseph never even knew whether he was dealing with a lone hacker or a group.

The CIO, Amanda Mitkin, was informed each time a break-in occurred. There seemed to be too many break-ins occurring on the network, and Amanda wanted to know why. Interestingly enough, the manager responsible for Global Chips's firewall complex didn't question the break-ins. He thought that the firewall administrator was a hero for fixing the problems.

Amanda's concerns were certainly valid. When a company installs a firewall, all traffic flowing from the intranet to the Internet (or vice versa) passes through the firewall. It's there for protection. Not target practice!

When executives at the top question why break-ins keep happening before anyone else does, it's easy to see that line management is not paying attention. Luckily for Global Chips, Amanda was appalled by the routine break-ins and demanded an investigation.

Day 1: Keeping The Bad Guys Out

Amanda ordered Perry Slone, the internal audit director, to investigate. Perry was also puzzled by the number of successful break-ins. Since Perry didn't have that type of expertise on staff, he hired a security consultant to conduct an audit.

That's where I came into the picture. Perry told me that the break-ins were becoming a routine event, but that was all he knew. Since he already knew how serious the situation was, I didn't have to spend time proving the level of risk. The hackers did that for me. Instead, the main goal of my audit was to answer his question, "Why can't we lock the hackers out?"

Perry passed me off to his right-hand man, Ted Davis. Ted was responsible for setting up meetings for me. Some managers and support people like to blow off auditors by not returning their phone calls, playing the hard-to-get routine. I didn't have the time (or inclination) to play that type of silly game. Ted's job was to make sure I didn't have to.

As a professional corporate auditor, Ted could easily reach people in high levels within the organization fast. He was responsible for setting up meetings and making sure I had access to the right people. This was a big company that wanted fast answers. Playing politics wasn't in the game plan. With Ted around to make that clear, I began to think about my approach.

Some audits consist mainly of interviews and of writing the final report. That sounds strange, but sometimes there is so much risk that it's staring you right in the face. Since the risk was already obvious to the CIO, I had a gut-level feeling that this was going to be one of those audits.

When a hacker can repeatedly walk through a firewall, there's usually an obvious problem like poor support, configurations, or products. Still, I wasn't sure how much testing I would need to do. I did know that the interviews would be key to gathering my data, however. I began to formulate a list of questions in my head.

Most of my questions were for the firewall administrator, Joseph Withers. After all, he was the person staying up nights trying to keep the hackers out. Ted scheduled a morning meeting for me with Joseph and asked him to bring the required documentation firewall policies, procedures, emergency response procedures, and a network diagram.

Ted also set up an interview with the support manager, Carl Sanchez. Since both meetings were scheduled for the next day, it looked like I wouldn't really get my feet wet until then. In the meantime, I decided to probe Ted for information.

Ted gave me background information about the company and the break-ins. Since time was critical on this assignment, I started writing my audit report. I usually write the audit report last, but I had enough information about the break-ins to start the background section of the report. Ted showed me to a visitor's office where I pulled out my laptop and started in. I completed the background section so I'd be ready to fill in the details right after my audit. (Right now, I could only guess and I don't get paid to guess. These guys wanted the facts, and I'd have those soon enough.) That was enough for one day.

Day 2: Firewall Administrator

Ted met me in the lobby and signed me in. He escorted me to Joseph's office. At our first meeting, Joseph seemed a little nervous. Of course, many people are nervous around auditors, so I tried to break the ice (with one of my good jokes). Joseph was unimpressed by my humor and definitely not interested in small talk. Writing off the soft approach, I asked him for the documentation I'd requested. He provided me with only the network diagram. When I asked for the rest of the documentation, he informed me, "I don't have firewall policies and procedures; it's not my job to write them. I know how the firewall is configured, and I know what to do when a break-in occurs."

Boy, was my assessment off. He wasn't nervous; he was arrogant! He figured that he knew how to support the firewall and that was enough. It was easy to see that he didn't understand the value of policies and procedures and that he saw me as a mere auditor (translate that to "nuisance").

Temporary Security

In addition to a bad attitude, Joseph had some unusual ideas about support techniques. Basically, the firewall had not been upgraded for several years, which is like leaving a rusty old lock on the door protecting the company's new chip designs, finances, human resources, and marketing information. It was easy to see how a hacker could rattle on that door and get in. Each time a hacker broke through the firewall, Joseph patched the system or installed a work-around. With that Band-Aid approach, the firewall was soon filled with so many plugged holes that it became difficult to manage and support. Although that may have spelled job security to Joseph, it spelled bad security to me.

I talked with Joe a little longer, and learned that Global Chips had a separate security organization to audit the environment, write some (but not all) security policies, and handle intrusions. Joseph was responsible for maintaining the firewall. Whenever there was a break-in, he paged the security guy on call. Joseph told me emphatically, "The security guys are responsible for security not me. It's their job to maintain the company security policies." It was pretty obvious that I would need to interview someone from the security group to get the other side of the story.

Joseph clearly fit into what I call the big-L category, and that's "L" for loser. Not only was he difficult to talk with, he was arrogant and an information holder. People who hold information rather than sharing it are very dangerous. They're typically insecure, and they think that the more information they keep to themselves, the more valuable they become. I didn't want to waste any more of my time with him. So, after my lovely interview with Joseph, I asked Ted to set up an interview with the company's security expert. I moved on.

Management and Security

My next step was to interview Joseph's manager, Carl Sanchez. Carl was dressed pretty casually in a golf shirt and jeans. However, he was one of those people who always manage to look dressed up regardless of what they're wearing. He had a nice smile and didn't seem to mind my sudden appearance into his world. This time, he was the one telling a joke to break the ice. All joking aside, I said, "Let's take a look at the serious break-ins that are occurring on your network. Carl, what do you think is going on?"

To my surprise, Carl didn't seem to think that the break-ins were serious. He seemed to be living in a dream world. He felt that he had one of the best firewall administrators in the world working for him. After all, Joseph did an excellent job of securing the firewall, and he knew exactly what to do when something went wrong.

I said, "You know, Carl, just because someone knows how to plug a hole, doesn't mean he has the knowledge to build a dam." No response from Carl. He probably didn't understand my comment.

I pressed on with my interview and informed Carl that they were operating the firewall without policies or procedures, but he already knew that. His position was that it wasn't his group's responsibility to write them. It actually took me quite a while to convince Carl that logically, his group was the only group that could write valid policies and procedures. Of course, I didn't really need to convince Carl that wasn't my job. But I'm really passionate about security and tend to get a little carried away. I feel very strongly that when people are being paid to maintain the door to a network, they should take it seriously!

I met briefly with the security expert, Frank Sarpa. Frank informed me that his group had never been asked to write policies and procedures for the firewall. Frank explained that his group wrote most security policies and procedures, but that Carl's group was responsible for the firewall. I asked him about the constant break-ins from the Internet. He said that they were in react mode most of the time and that they needed to design a new firewall complex to adequately protect the company from break-ins. He also added that his group made these suggestions to Carl over a year ago. Frank seemed like a really smart guy. He also seemed a little burnt out. I think he was tired of telling management what they should do, because they listened but never took any action.

It was clear to me that Global Chips had several problems. In auditing, I usually find more than one problem. Often, security problems are the result of a larger problem as in this audit. Since the roles and responsibilities weren't clearly defined, no one took responsibility for the firewall policies and procedures.

You can't have policies and procedures unless someone takes responsibility for writing and maintaining them. Obvious as that sounds, fixing this type of problem can be exceptionally difficult when the responsibilities for security cross organizational boundaries. In some organizations, the battle between divisions becomes more important than the data as at Global Chips. These guys didn't care who won the war, even if it was the hacker, so long as one of them won the battle. They took the battle more seriously than their real jobs of protecting the data.

Being Serious about Supporting Security

Firewall administration is a serious job. It should be taken seriously, by the administrator and by his or her manager. Global Chips didn't have the proper armor to survive threats against their data. Whenever a hacker breaks past a firewall, the attack could result in the modification, destruction, or theft of data. Casually patching up the problem and patting yourself on the back is not an adequate response, nor is management support of that type of behavior.

Carl didn't have a clue how risky it was to run an organization in react mode. He claimed that his division was in the process of designing a new firewall complex to replace the old one and predicted that they'd be ready to unplug the old firewall in six to nine months. He was very casual in his statements as if to say, "Don't worry about it. We have it under control."

We need people on the front lines who do worry! It was easy to tell that I was getting wrapped up in this audit. The passion switch needed to be turned off. I concluded interviewing for the audit. I could have spent another day testing the firewall, but everyone (who counted) agreed that it needed to be replaced because of the risk it posed to the network. I had enough data to put together the kind of report that executive management was expecting.

In my report, I identified a lot of security risks. The risks at the top of my list were:

• Roles and responsibilities were not properly defined.

• Firewall management and support were inadequate.

• Formal policies and procedures did not exist.

Since I'd gotten an early start in writing my final report the day before, filling in the details was fairly easy. I put in a few more hours on the report, then took off for the hills. Driving home, I felt pretty drained. It's always easier to deal with computers than it is to deal with humans. Humans are so complex sometimes too complex!

By the time I got home, the sun had fallen behind the hills. I took the rest of the night off and didn't think about the firewall. Not even once.

My Last Day: Attitudes Can Tell A Lot

The alarm clock went off before I knew it. Some days, 4:30 a.m. comes much too fast. It took me only a few minutes to focus on the day's planned events, including finishing my work with Global Chips.

That was enough to get me moving. I pulled myself out of bed and decided to run on my treadmill instead of going to the club. Finishing my run, I showered quickly, threw myself together, and split.

Driving back into Silicon Valley, I couldn't help think about how blind Carl was to believe that his firewall expert was doing him a favor. How do people like Joseph fool people like Carl so easily? Are managers like Carl really that nonchalant and uncaring about the data they support? Or, do they convince themselves that everything's OK because they don't want to think about the alternative? I guess those are some of the questions that even auditors can't answer.

It took me only a few hours to finish up the report. My meeting with Perry was at 3:00 p.m., and I was ready. Promptly at 3:00, Ted and I walked over to Perry's building to present my report. He was waiting for us.

As we quickly went through the report, I watched Perry's expression. It was one of disbelief. Oh, he believed the report. But he was astounded that the real risk boiled down to roles and responsibilities. Imagine, your entire company being at risk because the security roles and responsibilities were not clearly defined.

I didn't say anything about Carl and Joseph in the report. I never put that kind of information in writing anyway it's best left as a discussion point. I don't like to see people get fired, but sometimes it's one of the recommendations. I did mention that current employee attitudes would continue to put the company at risk, even when a new firewall was in place. The names and attitudes unsaid were clearly understood. My job was done.

Summary: Ask Not What Your Company's Security Can Do for You

It's frightening to see what can happen to a company when the proper roles and responsibilities aren't clearly defined. When it comes to supporting security, the "It's not my job" attitude can spread like wildfire. In this scenario, Global Chips was actually very lucky. Each hacker who broke into their system could have stolen chip designs or other critical data. Thus, that single firewall administrator had the potential to destroy the future of the entire company.

Of course, Joseph's inaction because "It wasn't his job" was only part of the problem. Managers like Carl exacerbate those problems by covering up the facts instead of taking responsibility for their territories and building solid support teams.

When people take responsibility for systems, they also inherit responsibility for the data on those systems. These guys didn't seem to realize that. I got the impression that Carl was more interested in golfing and the San Jose Sharks than in protecting the company. I like golf and I'm a Sharks fan too, but I take home a paycheck for doing my job, not pretending to. Carl (and Carl's manager too!) should have known that Joseph was producing bogus results.

Executive managers often detach themselves from what's really happening on the front lines. In this case, however, it was executive management that was concerned about the number of break-ins. It was the line-level managers who were apparently asleep at the wheel.

For security to work, every level of management must take responsibility for security. If executive management doesn't fund security, security suffers. If line management doesn't take an active approach to supporting security, security suffers. If middle management doesn't pass information to upper management, security suffers. Don't let security suffer in your company. Make sure that everyone understands what his or her role is.

As Marcus Ranum notes in his Internet Firewalls FAQ, "The Internet, like any other society, is plagued with the kind of jerks who enjoy the electronic equivalent of writing on other people's walls with spray paint, tearing their mailboxes off, or just sitting in the street blowing their car horns... A firewall's purpose is to keep the jerks out of your network while still letting you get your job done." And, it's truly amazing the number of jerks who are out there!

Even scarier, many of those delinquents have graduated from annoying vandalism to outright crime. Who's affected? Nearly everyone. CSI's 2002 study learned that 90 percent of respondents found computer security breaches. Although political entities (like the White House) and firms perceived as schoolyard bullies (like Microsoft) are obvious targets, all businesses face substantial risk. Even Web site defacement attacks, once viewed more as pranks than "real" crimes, actually come at a high price. Experts place the cost of security-related downtime for U.S. businesses at $273 billion a year. Businesses connected to the Internet are particularly at risk. While the Internet opens a door for businesses to expand into the global economy, it opens many windows of opportunity for computer crime as well. Closing those windows requires clear and consistently defined roles and responsibilities, as well as gateway technology like firewalls.