Encryption Schemes

Encryption is the process of transforming regular, readable data, or plaintext, into "scrambled" or unreadable form, called ciphertext. Decryption is the reverse process, the transforming of ciphertext into plaintext. The process of encryption can be used in various ways to ensure privacy, authenticity, and data integrity:

• Privacy No one should be able to view the plaintext message except the original sender and intended recipient.

• Authenticity The recipient of an encrypted message should be able to verify with certainty who the sender of the message is.

• Data Integrity The recipient of the message should be able to verify that it has not been tampered with or altered in any way while in transit.

Encryption is accomplished using an encryption algorithm, typically a pair of closely related mathematical functions that perform the actual encryption and decryption on the data provided to them. Modern encryption algorithms, including the ones used in Check Point NG, use what is called a key (or keys) to aid in the encryption or decryption process. There are two types of encryption algorithms: symmetric and asymmetric.

Encryption Algorithms; Symmetric versus Asymmetric Cryptography

In what is called symmetric encryption, the encryption algorithm itself is public, while the key is a secret. Anyone discovering the key, with knowledge of the algorithm, can decrypt any messages encrypted with that key. Since both the sender and recipient need to know the secret key before they can communicate, you must have a secure method of exchanging the key. Sometimes you will hear the term "Sneaker Net" used to describe this key exchange process, meaning that the exchange takes place via phone, fax, or in person, since an online exchange cannot be encrypted prior to the sharing of the key. Sometimes you will hear this key referred to as a "shared secret." Symmetric encryption is typically very fast, but has some disadvantages:

• As stated previously, anyone discovering the secret key can decrypt the messages.

• Since each sender-recipient pair (we will call them "users") needs a separate secret key, the number of separate keys that need to be managed increases rapidly as the number of users increases. Mathematically, we need n(n–1)/2 keys for a network of n users. Using this formula, a network of 500 users requires 124,750 unique keys.

Asymmetric encryption was developed to solve the problem of secure key exchange and to improve key management. It is called asymmetric because the encryption and decryption keys are different. In one form of asymmetric encryption, called "public key" encryption, both the sender and recipient each have two keys, one that is public and can be openly shared, and another that is private and is kept secret and never shared. If Alice wants to send an encrypted message to Bob, she and Bob only need to exchange public keys. The method used for the exchange need not be private in this case. Alice encrypts the plaintext message to Bob using Bob's public key. When Bob receives the message, he decrypts it using his private key. This method of public key encryption was invented in 1976 by Whitfield Diffie and Martin Hellman, and is sometimes called the "Diffie-Hellman" algorithm.

Another form of asymmetric encryption, called RSA encryption, is used by Check Point NG for generating digital signatures.

As we can see, asymmetric encryption solves the problem of key exchanges needing to be done in private. Users need only share their public keys to encrypt messages to one another. Asymmetric encryption does suffer one serious drawback, however: It is much, much slower than symmetric encryption (on the order of 1000 times slower). For this reason, real-life encryption schemes tend to use a hybrid form of public key exchange and private (symmetric) key encryption. Check Point NG is no different in this regard. A Diffie-Hellman key pair is used to generate and exchange a shared secret key, which is used for all encryption and decryption after the initial public key exchange. The shared secret key in this case is sometimes called a session key. The shared key can be regenerated at periodic intervals to lessen the chance of its compromise.

An encryption algorithm's security is completely dependent on its keys and how they are managed. Strong encryption that has a flawed key management algorithm is really weak encryption. You will often hear of an encryption algorithm described as using a 128-bit key, for example. What this means is that, if implemented properly, someone who tried to enumerate every possible key in order to break your encryption (called a brute force attack) would have to try 2128 different key combinations to be guaranteed success. This is not computationally feasible for the foreseeable future. In practice, cryptanalysts will typically attack an algorithm's key generation or key management scheme instead, attempting to find a flaw such as a predictable sequence of keys to exploit. The moral of all this is to pay attention to an algorithm's implementation, rather than to its key size exclusively. The latter will not guarantee your security. Note that asymmetric encryption schemes typically have key sizes that are much larger than symmetric ones (1024 bits, for example). The strength of these keys cannot be equated to the strength of symmetric keys, as they use different mathematical principles. The original Diffie-Hellman public key scheme, for example, was based on the difficulty of factoring very large prime numbers.

Check Point makes available several encryption algorithms. They are enumerated in Table 16.1, along with their shared key sizes and whether or not they are based on a public standard or are proprietary.

Table 16.1: Check Point Encryption Algorithms

Algorithm	Key-Length in Bits	Standard
FWZ-1	40-bits	Check Point proprietary
CAST	40-bits	Public
DES	56-bits	Public
3DES	168-bits	Public
AES	256-bits	Public

Key Exchange Methods: Tunneling versus In-Place Encryption

The previous encryption algorithms can be used in one of two key exchange schemes in FW-1: IKE (ISAKMP) or FWZ.

The Internet Security Association and Key Management Protocol (ISAKMP), or Internet Key Exchange (IKE), is an Internet encryption, authentication, and key exchange standard put forth by the IETF. It is widely used in today's Internet when implementing VPNs. Because it is a standard, a Check Point firewall using it will be able to interoperate with other third-party VPN products. We have tested or seen in production Check Point firewalls that interoperated with Linux gateways (Free/SWAN), OpenBSD, SonicWall, and Watchguard firewall products, as examples. The ISAKMP key exchange process is divided into two phases, and uses what are called Security Associations (SAs) to facilitate encryption and key generation. Keys and SAs are regenerated on a periodic basis.

IKE uses what is called tunneling-mode encryption. This means that each packet that is to be sent over a VPN is first encrypted (both header and data payload are encrypted), and then encapsulated with a new header. The new header will differ based on whether the packet is just being encrypted, just being authenticated, or both. This tunneling mode slightly degrades network performance, but is more secure.

FWZ is a Check Point proprietary key exchange scheme that uses another proprietary protocol, RDP (Reliable Datagram Protocol, not the same as the one described in RFC1151) to negotiate encryption and authentication methods between gateways.

FWZ uses what is called in-place encryption, in which packet bodies are encrypted, leaving the original TCP/IP headers in place. This method of encryption is faster than tunneling mode, but at the expense of security, since original header information is left in a readable state, including IP addresses, which are internal to an organization. Note that because FWZ does not encapsulate packets before sending them through a VPN, FWZ cannot be used in situations where any networks participating in the VPN domain have nonroutable addresses.

Hash Functions and Digital Signatures

A hash function, also known as a one-way function, is a mathematical function that takes a variable-length input and generates a fixed-length output, which is typically much smaller than the input. If we pass a plaintext message through a hash function, we produce what is called a message digest. A good hash function is one that, if we are given the message digest, it is impossible to reverse the function and deduce the original message. It is also one in which for any two different function inputs (two different messages in this context), the output should be unique to the input. To put it another way, the message digests for two different messages should also be different. As we will see in the following example, this principle can be used to ensure the integrity of a message. If a hash function generates the same message digest for two different inputs, we call this a collision. A good hash function will minimize collisions. When we talk about hash functions, we usually specify the length of the message digest in bits. This roughly corresponds (strengthwise) to the length of a symmetric encryption key. For example, a commonly used hash function, MD5, produces a 128-bit message digest for any size input it is given.

The output to a hash function is usually much smaller than the original message as well. MD4 and MD5 are good examples of hash functions. You may have heard of an MD5 checksum before. This checksum would be the result of sending a file through the MD5 hash algorithm.

Another important note about hash functions is that the output is unique to the message. If the original message was tampered with in any way, then a different message digest would result. Since you cannot "decrypt" a message digest, you run the algorithm against the message and compare the two digests to verify that the message is intact. This is how data integrity is achieved.

A digital signature is an attachment to a message that uses a hash function and enables the receiver to authenticate the sender and verify data integrity. Digital signatures can be attached to encrypted messages. Check Point NG generates digital signatures using an RSA private key and a hash function, as follows (assume that Alice wants to send a digitally signed message to Bob):

1. Alice sends the (unencrypted) message through a hash function, producing a fixed-length message digest.

2. Alice encrypts the message digest with her private RSA key, and sends it on its way, along with the encrypted message. The encrypted message is now "signed" by Alice.

3. Bob decrypts the message as usual, and passes it through the same hash function Alice used when it was sent. Bob compares this message digest he just generated with the decrypted message digest sent to him, making sure they match. Alice's public key is used to decrypt the message digest in this case.

A match in this case means that Bob can be sure that Alice sent the message, and that no one tampered with it in transit. We are assuming here that Bob trusts that he is using Alice's public key; this trust is usually provided by a certificate authority (CA) who will certify public keys.

The two hash functions offered by Check Point are MD5 and SHA-1. MD5 is a 128-bit hash function, while SHA-1 is considered more secure with a 160-bit message digest length.

Certificates and Certificate Authorities

A CA is a trusted third party that we can obtain a public key from reliably. A certificate is issued by a CA, and contains reliable information about the entity wanting to be certified as authentic. This could be a person's or firewall's public key, or a secure Web-server host name and domain.

In the case of Check Point NG VPNs, certificates can be used by encrypting gateways to exchange public keys and to authenticate one another. Typically, the gateways themselves or (in the case of FWZ) the management consoles act as CAs in this regard.

Types of VPNs

There are logically two types of VPNs: site-to-site and client-to-site. Site-to-site VPNs are what we normally think of when we think VPN—two gateways separating an insecure network (usually the Internet), with encrypted traffic passing between them.

Client-to-site VPNs, on the other hand, have a fixed gateway at one end and a mobile client on the other, perhaps with a dynamic IP address. This type of VPN is implemented by Check Point's SecuRemote or Secure Client products.

VPN domains

We can define a VPN domain as a group of hosts and/or networks behind a firewalled gateway that participate in a VPN. In a site-to-site VPN, each gateway has its own VPN domain defined, and is also aware of the other gateway's VPN domain. Any traffic coming from one VPN domain and going to the other (behind the opposing gateway) will be encrypted outbound, and then decrypted inbound at the other end.

VPN domains are defined on each gateway's firewall object, and must be set up with certain rules in mind. We talk about this in more detail when we discuss VPN implementation.