Deal with Security Later

Three years ago, George Marckowicz and Nathan Linski were two of the best staff engineers in the business. They were also a bit anxious about the long-term issues of job security, upward mobility, and becoming rich working for a big company. Like many others, they resolved those issues by striking out on their own.

George and Nathan named their start-up company TransWorld Internet Services. Their niche was providing low-cost, high-quality access to the Internet and maintaining data storage for their clients. Basically, TransWorld allowed ordinary home users to connect to the Internet and easily store information, without having to worry about backing up the data.

Unfortunately, they did not think of everything like configuring security. They connected their systems to the Internet out-of-the-box without configuring security, which left their systems and their customers' data wide open to attack.

Day 1: False Sense of Security

George and Nathan's ability to time the market was nearly as good as their technical backgrounds. Within six months, TransWorld was maintaining home directories and Web pages for over 1,000 customers.

Like any other new entrepreneurs, George and Nathan were worried about controlling costs. Luckily, their technical expertise allowed them to handle most of the hands-on functions on their own. For example, they had no problems installing all the systems and software themselves.

Unfortunately, George and Nathan configured all their systems out-of-the-box without considering additional security measures. They didn't seem to give security a second thought (or even a first thought). That's not unusual, because most engineers don't like security. Engineers thrive on easy access to information and tend to view security as an obstruction. But in a company that's responsible for maintaining the reliability and integrity of customer data, that's a major-league problem.

Two Years Later: Noticed the Attack

For George and Nathan, it was about two years before a hacker wandered into their site from the Internet. Nathan discovered the hacker because of a program he wrote that tracked his customers' access time (it was a pretty good program, too). The output of the program told Nathan how much to bill his clients for access time. Nathan noticed that once on TransWorld's intranet, the hacker created a directory, installed security tools (for collecting passwords, covering tracks, etc.), then began cracking system passwords and looking for data and access to other systems. (For an inside view of this type of attack, see Chapter 12, "A Hacker's Walk Through the Network.")

Nathan finally figured out that the hacker got in by using a login account left over from an old trade show. He disabled the account and considered the problem solved.

+ Two Weeks: The Hacker's Back

Two weeks later, the hacker resurfaced in the TransWorld network. This time, the hacker copied a program to their server that when executed, would mail the password file to another system on the Internet. At first, Nathan couldn't figure out how the hacker entered his network. After further investigation, Nathan discovered that he had installed the system so that anyone could copy and execute files from the Internet to his server. The hacker simply copied the program to TransWorld's server and executed it.

Upon execution, the program copied TransWorld's password file and mailed it to an unknown system on the Internet. Then the hacker cracked a password. Bingo! The hacker was in. Nathan's program to track system usage was good, but it was not a replacement for audit logs.

Since auditing was not configured, TransWorld staff couldn't even tell whether the hacker had replaced any system files or left back doors into the system. Nathan plugged the holes as best he could and tightened up the file permissions on the system.

+ Three Weeks: Fixing Security

The break-ins continued. Time after time, the hacker (or hackers!) simply strolled in from the Internet. Each time Nathan plugged up one security hole, the hacker quickly found another.

By this time, the TransWorld staff were all running in react mode. They were spending so much time reacting to the constant break-ins that they were beginning to lose sleep from fear of a real disaster.

At this point George called me for help. Since George and Nathan were old friends of mine, I agreed to check out their systems in exchange for a good chat and a couple of cold beers.

Knowing how much these guys knew about systems, I assumed that I was looking at a relatively quick fix. After all, they both had gobs of experience in operating and supporting systems and were maintaining a real ISP network. I knew that experience wasn't always a good indicator of security awareness, but I wanted to believe that they knew what they were doing. After all, this was an ISP responsible for customer data. I wanted to believe that they would take the necessary precautions.

In retrospect, maybe I was just fantasizing about what good security practitioners these guys were. If so, I guess I need to improve my fantasy life.

Since I thought I was looking at a quick fix, I told Nathan I'd stop by on my way into work. Nathan told me that the last break-in had occurred on his home network, so I decided to start there. First, I asked Nathan why his home network was connected to TransWorld. Nathan explained that he kept his development software on that system and needed easy access when he was working at their main headquarters.

That answer told me that this would not be a quick fix. Often, "easy access" is spelled "risk." In dealing with electronic information, you always need to weigh risk against business need. If security controls are too tight, you can obstruct your clients' ability to conduct business. Obviously, no one wants that. Any business needs some flexibility in order to function. Some companies, however, don't weigh what that flexibility costs in terms of risk. Instead, they simply shoot for easy access and then hope that the data's secure enough. As I drove to Nathan's house, I couldn't help wondering just how much Nathan valued his easy access.

I arrived at Nathan's house at 6:30 in the morning. Nathan offered me a cup of coffee. I was pumped up my adrenaline was already flowing. "No thanks," I said, "Where's the system?" He pointed me to the keyboard.

Within five minutes of logging in, I'd discovered what the security problem was. There was no security! The systems were installed out-of-the-box and connected to the Internet.

One of the biggest risks with out-of-the-box installs is that patches don't get applied. All operating systems have security vulnerabilities. Security patches must be applied to fix these problems; otherwise, the systems could be left wide open (depending on the vulnerability).

As I continued to look at the setup of the system, I was shocked at my findings. They were exporting home directories over the Internet with read/write permissions (to the world). I couldn't believe my eyes! Exporting filesystems over the Internet with read/write permissions allows anyone on the Internet to read, steal, or destroy the data. What were these guys thinking? I checked again and again, as if the results might be different the next time I checked, because I didn't want to believe what I saw.

It never occurred to me that two guys who could code circles around me and who charged customers for Internet access and data storage would set up a network without configuring any security at all!

As I tried to investigate further, however, Nathan kept asking me questions. I couldn't concentrate. My mind was running in circles and everything was blurred. I had to stop my audit. I turned to Nathan and told him I had to leave.

He had serious risks and I did not have a quick fix. That's what he really wanted a quick fix. Unfortunately, he wasn't going to get that from me (or from anyone else on this planet). Instead, I gave Nathan a list of emergency fix now items and told him I'd get back to him.

After I'd had a chance to think it through, I called back to let him know that there were so many security problems on his home network that I hardly knew where to tell him to start. My best advice for him was to conduct a full-scale security audit on the network. I told him that I was really busy working at Sun, helping customers secure their networks from the Internet, and that it would take a few weeks before I could come back. Besides, I was not going to have time to help him secure his mess; I could only report my findings to him. I suggested that he hire someone to help him audit and secure his network.

Nathan said he didn't mind waiting. A few weeks didn't seem like a big deal to him. Although he was obviously concerned about security, Nathan wasn't yet scared enough to hire someone to fix it. I wondered how much his customers cared that he was risking the reliability and integrity of their data.

The Saga Continues: The Network Remains at Risk

A few weeks later, I went to TransWorld's main headquarters to check things out. The first problem I noted in the full-scale audit was physical security. The network was located in a building that shared office space with several other companies. Only five-foot-high room dividers separated their servers, PCs, firewall, and network connections from the rest of the companies. The systems weren't even locked to the desks.

Furthermore, all of the customer addresses and accounting information were on the PCs. The PCs were backed up, but the backups were sitting right on the desk. Anyone who really wanted to could hop over the room dividers, pick up a system and its backups, and just walk away. Wouldn't that be a mess? Imagine your ISP calling you to find out if you paid your bill last month!

Data security wasn't much better. I already knew that, but you always need the hard facts. I positioned myself at a keyboard and started my audit. Within seconds, I had broken root and gained full control of their main server. It wasn't even challenging; I was able to exploit an operating-system security glitch that had been widely known for many years.

I usually look for those sorts of bugs when I try to break root on a system. If I can break root in just a few seconds, I typically know what I'm in for. And usually, it's serious risk that I find.

Pressing on with my audit, I found exactly what I thought I would find. Security wasn't configured here either. These systems were installed out-of-the-box without configuring security or adding additional controls. Basically, the problems were the same as for Nathan's home network (no surprise!).

Again, there were a ton of security vulnerabilities. Leading the risk list for TransWorld were the following items:

• Excessive file permissions existed.

• Old user accounts had not been deleted.

• Security patches had not been applied.

• Inadequate physical security existed.

Just these items alone spelled a very large potential for disaster. TransWorld needed at least two weeks of full-time security consulting just to begin to secure their network. Like most start-ups, however, they didn't want to spend their money on head count for security. I think they wanted to buy more equipment instead of course, maybe they wanted to get a paycheck that month too.

Anyway, I transferred the information to George over that promised beer. As for the network's security, I didn't give it much hope. George barely had time to talk over his beer. He was too busy trying to run the network, support customers, write code, and have a social life. Somehow, I suspected that being serious about security wasn't likely to happen any time soon.

Summary: Would You Hire This ISP?

Bad as the situation sounds, George and Nathan were really good people. Obviously, they knew how to install and support systems. They also knew the nuts and bolts of connecting to the Internet. Their problems were that they didn't know how to secure their systems and they didn't seek outside assistance. They also didn't really consider the possibility that a hacker would break into their network. Since they didn't view security as a priority, they allocated the available funding elsewhere. And, of course, you get what you pay for.

Actually, George and Nathan may have reason to worry in the future. The ISP/client relationship involves an implicit understanding that the ISP storing your Web page and home directory is responsible for the safety, reliability, and integrity of your data. To date, no case involving a negligent ISP who lost customer data has gone to court. So, it's hard to say which way a judge would rule. My guess is that the judge would assume (as most clients do) that since you pay a fee for the service, your data is in the hands of experts who should care about security and the safety of your data. After all, isn't that what you are paying for?

So far, the customers of TransWorld have had luck on their side. The funny thing is that they have no idea that their data is sitting on the edge of a cliff. It could fall off, never to be seen again. George and Nathan have remained lucky, too. At least to my knowledge, they haven't been sued or stalked by any of their clients. Of course, I still wouldn't use them for my own ISP.