Open SSH 2 and rsync

The rsync configuration described in this chapter will send all data through an Open Secure Shell 2 (henceforth referred to as SSH) connection to the backup server. This will allow us to encrypt all data before it is sent over the network and use a security encryption key to authenticate the remote request (instead of relying on the plaintext passwords in the /etc/passwd file).

A diagram of this configuration is shown in Figure 4-1.

Figure 4-1: rsync and SSH

The arrows in Figure 4-1 depict the path the data will take as it moves from the disk drive on the primary server to the disk drive on the backup server. When rsync runs on the primary server and sends a list of files that are candidates for synchronization, it also exchanges data block signatures so the backup server can figure out exactly which data changed, and therefore which data should be sent over the network.

SSH: A Secure Data Transport

Because we will push data from the primary server to the backup server, we will configure the primary server as an SSH client and then turn the backup server into an SSH server capable of accepting this data. The SSH connection between these two servers will need to enforce the following security configuration:

• When the primary server initiates a connection to the backup server, it should verify that it is in fact talking to the true backup server (and not a bogus host created by a hacker trying to hijack our data).

• The backup server should only accept connections from the primary server. In fact, the backup server should only accept connections from one trusted account on the primary server.

• All data sent over the network connection should be encrypted.

Using these security features, our data replication method can also be used on an untrusted public network such as the Internet.

This security configuration will also need to be fully automated (we should never need to walk over to the system console and enter a password to get the SSH connection working), and it should survive system reboots.

We can accomplish all of these security configuration goals using SSH encryption keys that allow us to establish a two-way trust relationship between the client and the server.

SSH Encryption Keys

Encryption keys, or asymmetric encryption keys (to be exact), always have two halves: a public half and a private half. The private half should be zealously guarded and never transmitted over the network. The public half should be given to anyone who wants, or needs, to send encrypted data. For SSH to work properly, we need to configure two types of encryption keys: the SSH host encryption key and the user encryption key.

The Host Encryption Key

The host encryption key is configured once for each SSH server and stored in a file on a locally attached disk drive. Red Hat stores the private and public halves of this key in separate files:^[2]

 /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.pub 

This key, or these key files, are created the first time the /etc/rc.d/init.d/sshd script is run on your Red Hat system. However, we really only need to create this key on the backup server, because the backup server is the SSH server.

The User Encryption Key

This key is created on the primary server manually by running the ssh-keygen utility. This key is stored in two files in the user's home directory in the .ssh subdirectory. For user web with a home directory of /home/web, the key would be stored in the following two files:

 /home/web/.ssh/id_rsa /home/web/.ssh/id_rsa.pub 

The public half of this key is then copied to the backup server (the SSH server) so the SSH server will know this user can be trusted.

Session Keys

As soon as the client decides to trust the server, they establish yet another encryption key: the session key. As its name implies, this key is valid for only one session between the client and the server, and it is used to encrypt all network traffic that passes between the two machines. The SSH client and the SSH server create a session key based upon a secret they both share— called the shared secret. You should never need to do anything to administer the session key, because it is generated automatically by the running SSH client and SSH server (information used to generate the session key is stored on disk in the /etc/ssh/primes file).

Establishing a two-way trust relationship between the SSH client and the SSH server is a two-step process. In the first step, the client figures out if the server is trustworthy and what session key it should use. Once this is done, the two machines establish a secure SSH transport.

The second step occurs when the server decides it should trust the client.

Establishing the Two-Way Trust Relationship: Part 1

Let's examine how this two-way trust relationship is established in more detail. First, the client must decide whether it can trust the server; then, the server must decide whether it can trust the client. So Part 1 of establishing the two-way trust relationship is when the client computer determines whether it is talking to the proper server.

Should the SSH Client Trust the SSH Server?

The first step in building a secure SSH transport is initiated by the client. The client needs to know if it is talking to the right server (no one has hijacked the IP address or hostname of our SSH server). This is accomplished during the SSH key exchange processes.

Note 

If you need to reuse an IP address on a different machine and retrain the security configuration you have established for SSH, you can do so by copying the SSH key files from the old machine to the new machine.

The SSH Key Exchange Process

The SSH key exchange process starts when an SSH client sends a connection request on port 22 to the SSH server (as shown in Figure 4-2).

Figure 4-2: SSH client connection request

Once the server and the client make some agreements about the SSH protocol, or version, they will use to talk to each other (SSH version 1 or SSH version 2), they begin a process of synchronized mathematical gymnastics that only requires a minimum amount of data to pass over the network. The goal of these mathematical gymnastics is to send only enough information for the client and server to agree upon a shared secret for this SSH session without sending enough information over the network for an attacker to guess the shared secret. Currently, this is called the diffie-hellman-group1-sha1 key exchange process.

The last thing the SSH server sends back to the client at the end of the diffie-hellman-group1-sha1 key exchange process is the final piece of data the client needs to figure out the shared secret along with the public half of its host encryption key (as shown in Figure 4-3).

Figure 4-3: SSH server response

Note 

The SSH server never sends the entire SSH session key or even the entire shared secret over the network, making it nearly impossible for a network-snooping hacker to guess the session key.

At this point, communication between the SSH client and the SSH servers stops while the SSH client decides whether or not it should trust the SSH server. The SSH client does this by looking up the server's host name in a database of host name-to-host-encryption-key mappings called known_hosts2 (referred to as the known_hosts database in this chapter^[3]). See Figure 4-4. If the encryption key and host name are not in the known_hosts database, the client displays a prompt on the screen and asks the user if the server should be trusted.

Figure 4-4: The SSH client searches for the SSH server name in its known_hosts database

If the host name of the SSH server is in the known_hosts database, but the public half of the host encryption key does not match the one in the database, the SSH client does not allow the connection to continue and complains.

 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint[4] for the RSA key sent by the remote host is 6e:36:4d:03:ec:2c:6b:fe:33:c1:e3:09:fc:aa:dc:0e. Please contact your system administrator. Add correct host key in /home/web/.ssh/known_hosts2 to get rid of this message. Offending key in /home/web/.ssh/known_hosts2:1 RSA host key for 10.1.1.2 has changed and you have requested strict checking. 

(If the host key on the SSH server has changed, you will need to remove the offending key from the known_hosts database file with a text editor such as vi.)

Once the user decides to trust the SSH server, the SSH connection will be made automatically for all future connections so long as the public half of the SSH server's host encryption key stored in its known_hosts database doesn't change.

The client then performs a final mathematical feat using this public half of the SSH server host encryption key (against the final piece of shared secret key data sent from the server) and confirms that this data could only come from a server that knew the private half of the public host encryption key it received from the SSH server. Once the client has successfully done this, the client and the server are said to have established a secure SSH transport (see Figure 4-5).

Figure 4-5: The SSH client and SSH server establish an SSH transport

Now, all information sent back and forth between the client and the server can be encrypted with the session key (the encryption key that was derived on both the client and the server from the shared secret).

Establishing the Two-Way Trust Relationship: Part 2

We've described half of the process for establishing the two-way trust relationship between the client and the server. Now, for Part 2, we need to discuss the other half, which takes places on the SSH server.

Can the Server Trust the Client?

We have solved only half the problem of a two-way trust between the client and the server; the client trusts the server—now the server needs to decide whether it should trust the client.

This can be accomplished using either hostbased authentication or user authentication.

Hostbased Authentication

Historically, hostbased authentication was accomplished using nothing more than the client's source IP address or host name. Crackers could easily break into servers that used this method of trust by renaming their server to match one of the acceptable host names, by spoofing an IP address, or by circumventing the Domain Name System (DNS) used to resolve host names into IP addresses.

System administrators, however, are still tempted to use hostbased authentication because it is easy to configure the server to trust any account (even the root account) on the client machine. Unfortunately, even with the improvements SSH has introduced, hostbased authentication is still cumbersome and weak when compared to the other methods that have evolved, and it should not be used.

User Authentication

When using user authentication, the SSH server does not care which host the SSH connection request comes from, so long as the user making the connection is a trusted user.

SSH can use the user's normal password and account name for authentication over the secure SSH transport. If the user has the same account and password on the client and server, the SSH server knows this user is trustworthy. The main drawback to this method, however, is that we need to enter the user's password directly into our shell script on the SSH client to automate connections to the SSH server. So this method should also not be used. (Recall that we want to automate data synchronization between the primary and backup servers.)

User Authentication using SSH Encryption Keys

Fortunately, a method that is better than either of these two methods is available using SSH encryption keys. All we have to do is store the public half of a user's encryption key on the SSH server. When we try to connect to the SSH server from the SSH client, the server will send us a challenge (a random number encrypted with the public half of the user encryption key), and only a client that knows the private half of the encryption key will be able to decrypt this challenge.

The private half of the user's encryption key on the SSH client is normally also protected by a passphrase. A passphrase is just a long password (it can contain spaces) selected by the user that is used to encrypt the private half of the user's real encryption key before it is stored on disk. We will not use a passphrase, however, because the passphrase would need to be added to our shell script to automate it.

Note 

You can instead use the Linux Intrusion Detection System, or LIDS, to deny anyone (even root) from accessing the private half of our user encryption key that is stored on the disk drive. See also the ssh-agent man page for a description of the ssh-agent daemon.

Here is a simple two-node SSH client-server recipe.

^[2]You will also see DSA files in the /etc/ssh directory. The Digital Signature Algorithm, however, has a suspicious history and may contain a security hole. See SSH, The Secure Shell: The Definitive Guide, by Daniel J. Barrett and Richard E. Silverman (O'Reilly and Associates).

^[3]Technically, the known_hosts database was used in Open SSH version 1 and renamed known_hosts2 in Open SSH version 2. Because this chapter only discusses Open SSH version 2, we will not bother with this distinction in order to avoid any further cumbersome explanations like this one.

^[4]You can verify the "fingerprint" on the SSH server by running the following command: #ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub