30.4 IPSec, Diffie-Hellman, and Modular Arithmetic

   

IPSec allows us to configure a secure communication channel between multiple machines. This secure communication channel operates at Layer 3 (the IP layer) of the ISO seven-layer model. As such, it can provide a secure communication channel for all of the common network services, e.g., SMTP, ftp, telnet, and DNS, without those individual services securing their own communications.

As we have mentioned previously, in an ideal cryptographic world we would use secret keys between individuals (or nodes, in this case). We have seen the problem of distributing secret keys (who said more beer?). IPSec uses the Diffie-Hellman crypto-system because it's means of establishing secret keys between two individuals over what is effectively an insecure communication medium, i.e., an IP network. The mathematics involved in Diffie-Hellman is deceptively simple and beautifully elegant. For once, I will take a slight detour via some maths just to give you an idea of what is happening here.

30.4.1 The basics of Diffie-Hellman

The strange thing about Diffie-Hellman is that it neither encrypts data nor signs it. To some, it would seem a strange choice of crypto-system due to these failings. Diffie-Hellman is simply a means whereby individuals can agree on a shared, secret key even though their communications are performed in a public placeon an IP network (or the pub). Here's an idea of how it works.

Some of this stuff gets into modular math. We don't get into heavy-math here, but just as a recap for you folks who haven't been in math class for a bit, modular math is used in most public key encryption schemes because it can be used as a mechanism of converting one digit into a different digit simply, efficiently , and with no chance of reversing the result without knowing a secret key. Modular arithmetic can be simply (-ish) described as taking the result of the addition and dividing it by some integer n . We discard the integer result of the division but keep the remainder. Using n = 10, we can say that

3 + 9 = 12 (mod 10) = 1 remainder 2 = 2

In this way, we can see that if we use 9 as a constant and add it to a digit mod 10, we can convert it into another digit = 2 in this instance. To decrypt, we use the additive inverse of 9 (mod 10). The additive inverse is the number you add to x to get 0. In normal arithmetic the additive inverse of 9 would be 9. In mod 10 arithmetic, the additive inverse of 9 is 1: 9+1=10 (mod 10) = 0. To encrypt, we add 9 (mod 10), and to decrypt, we add 1 (mod 10). Let's take our example above:

2 + 1 = 3 (mod 10) = 0 remainder 3 = 3

In this example, we have decrypted the number 2 back to its original value = 3. The same ideas work for modular multiplication and modular exponentiation. Some of the trickier parts of this include finding the multiplicative and exponetiative inverse of large numbers . There are clever math tricks like Euclid's Algorithm to help here. We won't go into all that here; I hope you can see that what we have is a means of encrypting and decrypting digits using some fancy math footwork. Let's jump way ahead and look at the basics of Diffie-Hellman.

Alice and Bob select two values: a prime ( p ) which is about 512 bits ¹ in size and a generator ( g ) which has a specific relationship to p ; for every number n between 1 and ( p-1 ) inclusive, there is some power of g ( g^m ) that equals n mod p , but that isn't really important in understanding the basics. Both p and g are made public. Both Alice and Bob choose a large (atleast 512 bits in size) number at random (hence, the need for a Strong Random Number Generator). This we will call Alice and Bob's secret. We then get into the fancy math footwork:

• Alice's secret we will call : Alice _secret

• Alice computes :

• Bob's secret we will call : Bob _secret

• Bob computes :

• Bob and Alice swap the T values :

• Alice computes :

• Bob computes :

The crazy bit about all this is that they both come up with the same answer! Here's how:

No one else can compute in a reasonable amount of time even though they now have and . If they could work out Alice _secret from , then they will have broken the Fundamental Tenet of Cryptography and we would (i) have a genius in our midst and (ii) be in a whole lot of cryptographic trouble.

30.4.2 The problem with Diffie-Helman

As it stands, your head is probably quite sore and you want to lie down for a bit. Let's just mention a failing of Diffie-Hellman: It offers no means of authentication. How do Bob and Alice know that they are talking to the right person? All they are doing is swapping numbers and from there deciding on a secret key. What happens if we reintroduce (the bitter and twisted) Eve into this scenario? Alice has no means of proving that she is talking to Bob and, likewise, Bob has no means of proving that he is talking to Alice. Figure 30-8 shows what is known as a man-in-the-middle or a bucket-brigade attack.

Figure 30-8. Man-in-the-middle attack.

Bob can be talking to Eve who is decrypting the messages, thank you very much, and then re-encrypting them with the shared key used for talking to Alice and forwarding the message to Alice. A similar process can be envisaged for the return messages, with Eve eavesdropping on all communications between Bob and Alice. This is why Diffie-Hellman is commonly used in conjunction with some form of authentication scheme to verify the sender and the recipients. IPSec can use either public keys or preshared keys to authenticate the sender and the recipient: This is known as Primary Authentication. In this way, Primary Authentication can offer privacy, authentication, authenticity, and non- repudiation in distributing a shared, secret key . The secret key can then be used to encrypt and authenticate the actual data packets being transmitted between the individual nodes.

The current version of IPSec (A.01.07) supports the use of VeriSign PKI (public key infrastructure) or Baltimore UniCert certificates for Primary Authentication. A third alternative is the use of a preshared key where both nodes need to know a secret key in order to perform Primary Authentication. This is useful when you want to demonstrate and test the capabilities of IPSec when you are waiting for the legal process involving notaries, and so on, to verify your Certificates with the Certification Authorities.

Version A.01.07 has discontinued the support of Entrust Security Certificates for Primary Authentication.

30.4.3 Setting up IPSec

IPSec comes free with HP-UX 11i but is not installed by default. It is available to download free from http://software.hp.com - Security and Manageability. The current version (A.01.07) is supported only on HP-UX 11.11; if you want IPSec for HP-UX 11.0 or 11.04, you should download version A.01.05. We look at setting up an IPSec between two nodes in order to encrypt a single service; we encrypt telnet packets between two nodes. This requires us to do the following:

• Install IPSec.
  
  
• Configure the IPSec policies including the encrypting and authentication of IP packets.
  
  
• Configure the ISAKMP Main Mode policies.
  
  
• Import/Request certificates or configure preshared keys .
  
  
• Set up boot-time configuration.
  
  
• Start the IPSec daemons.
  
  
• Test a connection to a remote machine to ensure that Main Mode and Quick Mode SAs are established.
  
  

We go through these steps in turn .

30.4.3.1 INSTALL IPSEC

This is not a laborious process; it's simply a swinstall that requires a reboot. Ensure that you have the product installed on all nodes wishing to use IPSec:

One problem I did encounter was that on HP-UX 11i with version A.01.07, the swinstall failed when I didn't have the 11.11 ARPA Transport patch PHNE_28895 (or later) installed. This patch is not part of the downloaded software bundle. I had to download this patch and its dependencies first before installing IPSec.

Setting up IPSec is performed via a GUI interface. Before we can interact with the GUI, we must set up a password to secure the IPSec configuration. This is required in case someone compromises the security of the root user on this system. Without the password, anyone attempting to view IPSec information will not be able to determine how our security associations have been configured.

The IPSec password is unique to an individual host and needs to be a minimum of 15 characters long. It is established using the ipsec_admin np command.

 root@hpeos004[.root]  ipsec_admin -np  IPSEC_ADMIN: Establishing IPSec password, enter IPSec password: ******************* IPSEC_ADMIN: Re-enter IPSec password to verify: ******************* IPSEC_ADMIN: ALERT-IPSec password successfully established. root@hpeos004[.root] 

If you were to lose or forget this password or if you believe your IPSec configuration had been compromised, you would have to do the following:

• Remove the software.
  
  
• Remove the following files under /var/adm/ipsec :
  
  

  1. .admin_info

  2. .ipsec_info

  3. javabeans. info

  4. .lock

  5. cainfo.txt

  6. certs.txt

  7. entrust.txt

  8. pskeys.txt

• Revoke your certificates with your Certification Authorities.
  
  
• Reinstall the software.
  
  

30.4.3.2 CONFIGURE THE IPSEC POLICIES INCLUDING THE ENCRYPTING AND AUTHENTICATION OF IP PACKETS

We can now configure the IPSec policies. IPSec policies use the secure communication channels (Security Associations) established by the IKE (Internet Key Exchange) daemon: On HP-UX, this is known as the ikmpd daemon ( ikmpdv6 if you have IPv6 installed). This daemon deals with establishing Main Mode Security Associations . There are two other daemons: The secpolicyd deals with establishing Quick Mode Security Associations and has IPSec and ISAKMP policy information. It will receive a query about a 5-tuple (local IP address, remote IP address, local port number, remote port number, and protocol number) and find the best-fit IPSec policy. It will note whether an IPSec policy has been used to establish an IPSec SA, and record the Security Parameters Index (a SPI number is used to identify IPSec SAs) for outbound IPSec SAs. Note that the IPSec policies can contain IP address masks and wildcards. Therefore, a single IPSec policy can match multiple 5-tuples and result in multiple IPSec SAs. The secauditd receives audit messages from the other modules and logs them in an audit file. The two other major components of IPSec are the Kernel Policy Engine , which caches recent policy decisions, and SAs for specific 5-tuples. It will also record SPIs for outbound IPSec SAs. Finally, there is the Kernel Security Association Engine , which keeps a database of IPSec SAs, indexed by the Security Parameter Index (SPI) and IP addresses. This database contains the IPSec SA parameters, including the cryptography keys. Note that unlike IPSec policies, the cache entries do not contain masks or wildcards.

A Security Association (SA) is a secure communication channel and its parameters such as encryption method, keys, and lifetime of keys. As such, a single telnet session will have at least two SAs: an inbound and an outbound SA.

Main Mode is where the ikmpd daemon established the ISAKMP (Internet Security Association and Key Management Protocol) SAs. This includes authentication and encryption algorithms and key lifetimes. Next , the two nodes will exchange information used in the Diffie-Hellman algorithm. Finally, both nodes authenticate each other using the chosen Primary Authentication means (either public keys or preshared keys ).

Quick Mode relates to specific IPSec policies that include Filters and Transforms . Quick Mode SAs utilize the secure communication channel set up by earlier ISAKMP ( Main Mode ) SAs. By default, 100 Quick Mode SAs can utilize a single ISAKMP SA. A concept known as Perfect Forward Secrecy (PFS) allows the ikmpd daemon to create a new ISAKMP SA for every IPSec SA. This is computationally expensive, but it does mean that if a single key is compromised, only that one SA may be compromised. (We discuss ISAKMP Main Mode Security Associations later.)

An important part of the IPSec policy is the Filters we will use. Filters can include local/remote IP addresses, local/remote subnet masks, protocols, and even local and remote port numbers. This allows us to encrypt and/or authenticate individual services, e.g., telnet .

A second part of the IPSec policy is the Transform we will use to encrypt and/or authenticate individual packets.

30.4.3.2.1 IPSec Authentication Headers

To authenticate an IP packet, we include an additional field in the header of the packet. This header is known as an Authentication Header. The IP Authentication Header (AH) provides integrity and authentication but no privacy: The IP data is not encrypted.

The AH contains an authentication value based on a symmetric key hash algorithm. All the fields in the IP datagram that are not mutable (do not change in transit) are used to calculate the authentication value; this includes the IP Header as well as other headers and the user data. IP Fields or options that need to change in transit, such as hop count and time to live , are assigned a zero value for the calculation of the authentication value.

The AH header also includes a sequence number as protection against replay attacks. HP's IPSec checks the sequence number on received packets to prevent replay attacks.

Table 30-2. Authentication Header in Transport Mode.

Included in the AH header is a Security Parameter Index (SPI) value as well as the message digest. The SPI is a key into an in-kernel Security Association table that tells IPSec which algorithm was used to authenticate the packet. IPSec support two U.S. government standards for AH: HMAC-SHA1 (Hashed Message Authentication Code with Secure Hash Algorithm) and HMAC-MD5 (HMAC with Message Digest-5). Both will use the secret key established by both nodes during the Main Mode negotiations.

An example AH transform would be of the form: AH-MD5 .

30.4.3.2.2 IPSec Encapsulated Security Payload Headers

While the Authentication Header provides integrity authenticity, it does not provide privacy. To provide privacy, we must encrypt the packet as well. This is where the IPSec ESP (Encapsulated Security Payload) header comes in. This will add an additional field to the packet. Included in the ESP header is a Security Parameter Index (SPI), similar to the SPI value in an AH header, that will determine the algorithm used for encryption. The IP Encapsulating Security Payload takes the IP payload, such as a TCP packet, encrypts it using a symmetric key, and encapsulates it with header information so that the receiving IPSec entity can decrypt it. The ESP header also includes a sequence number as protection against replay attacks. All senders are required to properly set the sequence number, but receivers are not required to check it. HP's IPSec checks the sequence number on received packets to prevent replay attacks.

Table 30-3. Encapsulated Security Payload Header in Transport Mode.

IPSEC/9000 supports the 56-bit Data Encryption Standard Cipher Block Chaining Mode (DES-CBC) and Triple-DES CBC (3DES-CBC) encryption algorithms as well as the 128-bit Advanced Encryption Standard (AES-128). AES-128 provides encryption strength comparable to Triple-DES with the system performance comparable to DES.

An example ESP transform would be of the form: ESP-AES128 .

30.4.3.2.3 Authenticated or Nested ESP

To provide authentication and privacy, we can include an ESP authentication field at the end of the IP packet. With authenticated ESP algorithms, an authentication algorithm uses a second, shared secret key to compute an Integrity Check Value (ICV) that authenticates the ESP header and IP data. The ICV is appended to the end of the packet and can be generated using either HMAC-MD5 or HMAC-SHA1.

Table 30-4. Authenticated ESP.

30.4.3.2.4 Nested ESP

Alternately, we can nest an ESP transform within an AH. For example, you can nest a DES-CBC ESP transform within an HMAC-MD5 AH transform. IPSec will use DES-CBC to build an ESP with the IP data encrypted. IPSec will then nest this within an AH transform, using HMAC-MD5 to authenticate the ESP and the entire IP packet, including the immutable fields of the original IP header. This differs from an authenticated ESP transform using DES-CBC and HMAC-MD5. The authenticated ESP transform authenticates only the ESP header and IP data, and does not authenticate the original IP header.

An example of an Authenticated ESP transform would be of the form: ESP-DES-HMAC-MD5 .

Table 30-5. Nested ESP.

An example Nested AH & ESP transform would be of the form: AH-SHA1&ESP-AES128 .

30.4.3.2.5 Tunneling Mode for AH and ESP headers

Tunneling Mode is where we will include the original IP header in the AH and/or ESP algorithm. This is more relevant to ESP headers that will be encrypted. We generally use ESP in tunneling mode when IPSec is being used on a gateway machine. The additional, outer IP header will contain the IP address of the gateways, while the inner, encrypted IP header will contain the ultimate source and destination addresses.

Table 30-6. Tunnel Authenticated ESP.

Because IPSec will encrypt Layer-4 port numbers, it should not be protected behind a corporate firewall. The firewall may not allow packets through because it is may be filtering on Layer-4 port numbers, which are now encrypted.

Whichever type of transform we choose to use, we need to ensure that we use the same transform on both nodes involved.

30.4.3.2.6 Using the GUI to configure IPSec policies

The GUI supplied with the IPSec software is the mechanism we use to set up IPSec and ISAKMP policies. The command is ipsec_mgr . We are asked for the IPSec password before it launches the GUI:

 root@hpeos004[.root]  ipsec_mgr  IPSEC_MGR: Please enter the IPSec password: ******************* 

Once started, we are presented with the main IPSec Policy window. From here, we can c reate an IPSec policy. Figure 30-9 shows the steps I went through to create a single IPSec policy.

Figure 30-9. Creating an IPSec policy.

These bullet points relate to the steps I have taken:

1. Click Create to create a new IPSec policy. This opens the Edit IPSec Policy window.
   
   
2. Give the policy a name . I called my policy InboundTelnet ; the reason becomes obvious.
   
   
3. I chose to use Exclusive SAs. This flag allows you to configure SAs to be Shared (Host-based keying) or Exclusive (Session-based keying). If you check the Exclusive Flag, you are selecting Session-based keying. If not, the default (an unchecked box) is host-based keying. Host-based keying allows multiple connections (or sessions) between two systems to share the same two SAs created by IPSec. With session-based keying, a unique pair of SAs is used per connection or session, thus incurring a large overhead. It also, however, allows for a more secure or private connection.
   
   
4. I specify the local IP address (if required) to filter for.
   
   
5. I specify the remote IP address (if required) to filter for.
   
   
6. I decided to filter on specific TCP ports, so I clicked the check box for Configure Policy Based on Service .
   
   
7. I chose the specific port based on port name.
   
   
8. I chose the direction, i.e., inbound . Remember that an SA is a single secure communication channel; hence, TCP connections will require two policies, one inbound and one outbound .
   
   
9. I chose my Transform List by clicking the Edit button. This opens the Transform List window with combinations of AH and ESP. Here, I can choose which authentication and encryption algorithms I will use. My choice determines whether I use an AH, ESP, Authenticated ESP, or Nested ESP. I have chosen a Nested ESP. This was achieved by selecting the appropriate AH ( AH-SHA1 , in my case), keeping the ctrl key depressed and then selecting the appropriate ESP ( ESP-AES128 , in my case) and then clicking Add in the Transform List window. This resulted in the transform AH-SHA1&ESP-AES128 .
   
   
10. I finished my IPSec policy by clicking OK .
    
    

I need to repeat these steps to create an OutboundTelnet policy (obviously, the Direction field in Step 7 will be outbound in this case). I can choose a different Transform if I like; I decided to keep them the same, for simplicity.

NOTE : I need to ensure that I mirror this configuration on my remote host as well, especially the Transforms used for encryption/authentication .

30.4.3.3 CONFIGURE THE ISAKMP MAIN MODE POLICIES

If we want to stick with the default ISAKMP Main Mode policies, we need do nothing here. If, for instance, we want to establish Perfect Forward Secrecy, we need to c reate at least one ISAKMP policy. Figure 30-10 shows me setting up an ISAKMP policy for PFS (which really needs Exclusive SAs in your IPSec policies to be configured as I did above):

1. Click Create to open the Create ISAKMP Policy window.
   
   
2. Give the Policy a name; I called mine PFS for obvious reasons. Because I want this policy to apply to all ISAKMP Sas, I am leaving the Remote IP Address as an asterisk (*).
   
   
3. For Perfect Forward Secrecy, you need to set Max Quick Mode to equal 1 .
   
   
4. Click OK to save these settings.
   
   

Figure 30-10. Setting up Perfect Forward Secrecy.

PFS is computationally expensive because for every Quick Mode SA, a new ISAKMP SA is created. This should be undertaken only after careful thought. If you are seeking to establish PFS, then ensure that you configure a similar ISAKMP policy on all remote machines involved.

30.4.4 Import/Request certificates or configure preshared keys

Under Options on the menu bar, there is an option with a check box for Certificate Authority Settings . When checked, this activates the Certificates and Certificate IDs tabs in the main IPSec window. From there, you can request and/or import VeriSign and Baltimore certificates. I am going to choose to set up preshared keys . This is a secret key known to both nodes. The longer the preshared key, the more secure it is (see Figure 30-11). This is used to establish Primary Authentication when setting up the ISAKMP SAs. Ideally, I would use a public key, but time precludes such a setup; preshared keys are sufficient for testing and demonstration purposes.

1. Click Create to open the Create ISAKMP Preshared Key window.
   
   
2. Enter the IP address of the remote machine involved. You need to set up a preshared key for each machine with which you are communicating.
   
   
3. You can use a file as input; supposed you had a file containing a long key from another application, e.g., ssh-keygen , which produces a readable key file. I have decided to type in the key. I had to click on the input panel and then type the key by hand. Be sure that you type it correctly because you need to input exactly the same key on the remote machine.
   
   
4. Click OK to save your changes.
   
   

Figure 30-11. Setting up a preshared key.

I now need to mirror this configuration on my remote machine. NOTE: If using public key certificates, we should set up a cron job to periodically retrieve the Certificate Revocation List (CRL) from the appropriate servers. Two scripts come supplied with IPSec to retrieve the CRL from Baltimore ( baltimoreCRL.cron ) and VeriSign ( crl.cron ). The scripts can be found in /var/adm/ipsec_gui/cron :

 root@hpeos004[cron]  pwd  /var/adm/ipsec_gui/cron root@hpeos004[cron]  ll  total 2086 -r-x------   1 bin        bin            309 Feb  6  2003 baltimoreCRL.cron -r-x------   1 bin        bin            375 Feb  6  2003 crl.cron -r-x------   1 bin        bin        1065192 Aug  7 18:54 crl_cron root@hpeos004[cron] 

It is worthwhile to contact the relevant organization to establish how often they update the CRL in order for you to set up an appropriate cron job. It is not uncommon to run these jobs a number of times during the day. The scripts use the information configured in the ipsec_mgr program to contact the appropriate server and retrieve the CRL. The CRL is added to the certs.txt file.

30.4.5 Set up boot-time configuration

We need to ensure that IPSec is started at boot-time. This creates a file called /var/adm/ipsec/.admin_info , which is an encrypted version of the IPSec administration password that we created using the ipsec_admin np command. This is needed in order for IPSec to start up at boot-time without asking for the IPSec administration password. This is accomplished by accessing the Boot-up Options window as seen in Figure 30-12.

1. Access the Boot-Up Options window via the IPSec main window menu bar, under Options - Boot-up Options
   
   
2. Click the check box to Enable IPSec at Boot-up .
   
   
3. We need to fill in the file that holds our policy details. The default filename is /var/adm/ipsec/policies.txt .
   
   
4. Click OK to save the changes.
   
   

Figure 30-12. Configuring IPSec boot-up options.

We are now ready to start the IPSec daemons.

30.4.6 Start the IPSec daemons

We can see the status of the daemons using the ipsec_admin status command:

 root@hpeos004[.root]  ipsec_admin -status  ----------------- IPSec Status Report ----------------- Time:  Fri Oct 17 09:35:19 2003         secauditd program: Not running         secpolicyd program: Not running         ikmpdv6 program: Not running         IPSec kernel: Down         IPSec Audit level: Unknown         IPSec Audit file: Unknown         Max Audit file size: 0 KBytes         IPSec Policy file: Unknown         Level 4 tracing: None -------------- End of IPSec Status Report ------------- root@hpeos004[.root] 

To start the daemons, we use the same command but with the start option:

 root@hpeos004[.root]  ipsec_admin -start  IPSEC_ADMIN: Starting up the secauditd program. IPSEC_ADMIN: ALERT-Starting up IPSec/9000. IPSEC_ADMIN: Starting up the ikmpdv6 program. IPSEC_ADMIN: The ikmpdv6 program successfully started up. IPSEC_ADMIN: Starting up the secpolicyd program. IPSEC_ADMIN: Starting up the IPSec kernel. IPSEC_ADMIN: IPSec kernel successfully started up. IPSEC_ADMIN: Security Association Data Base successfully flushed. IPSEC_ADMIN: IKE MM SAs successfully flushed. root@hpeos004[.root]  ipsec_admin -status  ----------------- IPSec Status Report ----------------- Time:  Fri Oct 17 09:37:24 2003         secauditd program: Running and responding         secpolicyd program: Running and responding         ikmpdv6 program: Running and responding         IPSec kernel: Up         IPSec Audit level: Error         IPSec Audit file: /var/adm/ipsec/auditFri-Oct-17-09-37-18-2003.log         Max Audit file size: 100 KBytes         IPSec Policy file: /var/adm/ipsec/policies.txt         Level 4 tracing: None -------------- End of IPSec Status Report ------------- root@hpeos004[.root] 

I will start the daemons on all machines concerned .

30.4.7 Test a connection to a remote machine to ensure that Main Mode and Quick Mode SAs are established

The first time we try and perform a telnet between our two nodes in this configuration, the telnet will initially appear to hang while both machines establish the required Security Associations. Once established, the telnet session will behave as normal. In a separate session, I will monitor the Main Mode Security Associations that have been established ( ipsec_report mad ). I could also monitor the Quick Mode Security Associations ( ipsec_report sad ), although because I have established PFS, I may have to look at all active records ( ipsec_report ipsec active ) to see all currently active SAs.

The following commands were issued while a telnet session was active between node hpeos004 and hpeos003 . First, the Main Mode SAs:

 root@hpeos004[.root]  ipsec_report -mad  ------------------------ ISAKMP SA -------------------------- Sequence number: 1 Role: Initiator Local IP Address: 192.168.0.66 Remote IP Address: 192.168.0.65 Oakley Group: 1    Authentication Method: Preshared Keys Authentication Algorithm: HMAC-MD5    Encryption Algorithm: DES-CBC Quick Modes Processed: 0    Lifetime (seconds): 28800 root@hpeos004[.root] 

Now all active SAs:

 root@hpeos004[.root]  ipsec_report -ipsec active  --------------------- Active IPSec Policy Rule ----------------------- Rule Name: inboundtelnet      ID: 1 Cookie: 2    State: SPI(s) Not Established Src IP Addr: 192.168.0.66   Prefix: 32  Src Port number: 23 Dst IP Addr: 192.168.0.65   Prefix: 32  Dst Port number: * Network Protocol: TCP    Direction: outbound Action: Secure Shared SA: Yes Number of SA(s) Needed: 2 Number of SA(s) Created: 0 Kernel Requests Queued: 0 -- SA Number 1 --    SA Type: ESP    Encryption Algorithm: AES128-CBC    Authentication Algorithm: None    SPI (hex): Not Established    SPI updated: Unknown -- SA Number 2 --    SA Type: AH    Authentication Algorithm: HMAC-SHA1    SPI (hex): Not Established    SPI updated: Unknown --------------------- Active IPSec Policy Rule ----------------------- Rule Name: outboundtelnet      ID: 2 Cookie: 3    State: SPI(s) Not Established Src IP Addr: 192.168.0.66   Prefix: 32  Src Port number: * Dst IP Addr: 192.168.0.65   Prefix: 32  Dst Port number: 23 Network Protocol: TCP    Direction: outbound Action: Secure Shared SA: Yes Number of SA(s) Needed: 2 Number of SA(s) Created: 0 Kernel Requests Queued: 0 -- SA Number 1 --    SA Type: ESP    Encryption Algorithm: AES128-CBC    Authentication Algorithm: None    SPI (hex): Not Established    SPI updated: Unknown -- SA Number 2 --    SA Type: AH    Authentication Algorithm: HMAC-SHA1    SPI (hex): Not Established    SPI updated: Unknown --------------------- Active IPSec Policy Rule ----------------------- Rule Name: inboundtelnet      ID: 1 Src IP Addr: 192.168.0.65   Prefix: 32  Src Port number: * Dst IP Addr: 192.168.0.66   Prefix: 32  Dst Port number: 23 Network Protocol: TCP    Direction: inbound Action: Secure Shared SA: Yes Number of SA(s) Needed: 2 -- SA Number 1 --    SA Type: ESP    Encryption Algorithm: AES128-CBC    Authentication Algorithm: None -- SA Number 2 --    SA Type: AH    Authentication Algorithm: HMAC-SHA1 --------------------- Active IPSec Policy Rule ----------------------- Rule Name: outboundtelnet      ID: 2 Src IP Addr: 192.168.0.65   Prefix: 32  Src Port number: 23 Dst IP Addr: 192.168.0.66   Prefix: 32  Dst Port number: * Network Protocol: TCP    Direction: inbound Action: Secure Shared SA: Yes Number of SA(s) Needed: 2 -- SA Number 1 --    SA Type: ESP    Encryption Algorithm: AES128-CBC    Authentication Algorithm: None -- SA Number 2 --    SA Type: AH    Authentication Algorithm: HMAC-SHA1 --------------------- Active IPSec Policy Rule ----------------------- Rule Name: default      ID: 3 Cookie: 1    State: Ready Action: Pass Shared SA: No Number of SA(s) Needed: 1 Number of SA(s) Created: 0 -- SA Number 1 --    SA Type: AH    Authentication Algorithm: HMAC-SHA1    SPI (hex): Not Established    SPI updated: Unknown root@hpeos004[.root] 

At the same time, I was running a nettl trace on the telnet session to see if I could capture root passwords. Just to remind us of the commands I used, I will demonstrate the trace with IPSec turned off. Here, I start the trace on the IP layer using an output file /tmp/noIPsec.TRC000 :

 root@hpeos004[.root]  nettl -e NS_LS_IP -f /tmp/noIPsec -tn pduout pduin  

I then run the telnet session between nodes hpeos004 and hpeos003 . Once I log in, I can stop the trace:

 root@hpeos004[.root]  nettl -tf -e all  

I will use a formatting file just to filter for the source and destination IP addresses I am interested in:

 root@hpeos004[.root]  cat .netfmt.conf  filter ip_saddr 192.168.0.66 filter ip_daddr 192.168.0.65 root@hpeos004[.root] 

The next part of deciphering a network trace is knowing what you are looking for. In my case, I happen to know the format of the packets I am looking for. The packets I am looking for contain the clear-text passwords transmitted over the network. Consequently, I can use a netfmt command piped directly into a suitable grep command, looking for the packets of interest:

 root@hpeos004[.root]  netfmt -c .netfmt.conf -f /tmp/noIPsec.TRC000  grep '^   *32:.*[a-z0-9][a-z0-9] -- -- -- -- -- -- --  P.*'  more  ---------------------- SUBSYSTEM FILTERS IN EFFECT -----------------         ---------------- LAYER  1 -----------------         ---------------- LAYER  2 -----------------         ---------------- LAYER  3 -----------------         filter ip_saddr          hpeos004         filter ip_daddr          hpeos003         ---------------- LAYER  4 -----------------         ---------------- LAYER  5 ----------------- ---------------------- END SUBSYSTEM FILTERS -----------------------   32: 50 18 80 00 2b 2e 00 00 72 -- -- -- -- -- -- --  P...+...r.......   32: 50 18 80 00 2e 2c 00 00 6f -- -- -- -- -- -- --  P....,..o.......   32: 50 18 80 00 2e 2a 00 00 6f -- -- -- -- -- -- --  P....*..o.......   32: 50 18 80 00 29 28 00 00 74 -- -- -- -- -- -- --  P...)(..t.......   32: 50 18 80 00 35 18 00 00 68 -- -- -- -- -- -- --  P...5...h.......   32: 50 18 80 00 38 17 00 00 65 -- -- -- -- -- -- --  P...8...e.......   32: 50 18 80 00 31 16 00 00 6c -- -- -- -- -- -- --  P...1...l.......   32: 50 18 80 00 31 15 00 00 6c -- -- -- -- -- -- --  P...1...l.......   32: 50 18 80 00 2e 14 00 00 6f -- -- -- -- -- -- --  P.......o.......   32: 50 18 80 00 6c 13 00 00 31 -- -- -- -- -- -- --  P...l...1.......   32: 50 18 80 00 6b 12 00 00 32 -- -- -- -- -- -- --  P...k...2.......   32: 50 18 80 00 6a 11 00 00 33 -- -- -- -- -- -- --  P...j...3.......   32: 50 18 80 00 31 69 00 00 65 -- -- -- -- -- -- --  P...1i..e.......   32: 50 18 80 00 1e 67 00 00 78 -- -- -- -- -- -- --  P....g..x.......   32: 50 18 80 00 2d 65 00 00 69 -- -- -- -- -- -- --  P...-e..i.......   32: 50 18 80 00 22 63 00 00 74 -- -- -- -- -- -- --  P..."c..t....... root@hpeos004[.root] 

I hope that you can see that the root password for node hpeos003 is hello123 .

I performed the same trace, but with IPSec turned on. The trace file was called /tmp/IPsecON.TRC000 . I won't go through all the commands I used, because they were exactly the same as those used above. The part I am interested in is the content of the trace. Here's the result:

 root@hpeos004[.root]  netfmt -c .netfmt.conf -f /tmp/IPsecON.TRC000  grep '^   *32:.*[a-z0-9][a-z0-9] -- -- -- -- -- -- --  P.*'  more  ---------------------- SUBSYSTEM FILTERS IN EFFECT -----------------         ---------------- LAYER  1 -----------------         ---------------- LAYER  2 -----------------         ---------------- LAYER  3 -----------------         filter ip_saddr          hpeos004         filter ip_daddr          hpeos003         ---------------- LAYER  4 -----------------         ---------------- LAYER  5 ----------------- ---------------------- END SUBSYSTEM FILTERS ----------------------- root@hpeos004[.root] 

As you can see, the grep command has produced no output. At least I can see that something is different. This is where looking for individual packets becomes exceedingly difficult; the entire TCP data is now encrypted and bears no resemblance to what it used to look like. I even went so far as to look for packets containing the character I knew would be there:

 root@hpeos004[.root]  netfmt -c .netfmt.conf -f /tmp/IPsecON.TRC000  grep   '^....:.*  P.*[roothello123].*'  more  ---------------------- SUBSYSTEM FILTERS IN EFFECT -----------------         ---------------- LAYER  1 -----------------         ---------------- LAYER  2 -----------------         ---------------- LAYER  3 -----------------         filter ip_saddr          hpeos004         filter ip_daddr          hpeos003         ---------------- LAYER  4 -----------------         ---------------- LAYER  5 ----------------- ---------------------- END SUBSYSTEM FILTERS -----------------------   32: 50 10 80 00 ee 68 00 00 -- -- -- -- -- -- -- --  P....h..........   32: 50 18 80 00 9c 31 00 00 46 15 00 05 03 80 00 49  P....1..F......I   32: 50 10 80 00 e7 68 00 00 -- -- -- -- -- -- -- --  P....h..........   32: 50 18 80 00 95 65 00 00 46 15 00 05 03 80 00 49  P....e..F......I   32: 50 18 80 00 c6 a2 00 00 12 02 00 06 02 00 00 31  P..............1  240: 20 20 20 50 61 6c 6f 20 41 6c 74 6f 2c 20 43 41     Palo Alto, CA   32: 50 18 80 00 91 b2 00 00 72 6f 6f 74 40 68 70 65  P.......root@hpe   32: 50 18 80 00 e8 68 00 00 4c 05 00 06 03 00 00 49  P....h..L......I 1248: 50 73 65 63 4f 4e 2e 54 52 43 30 30 30 6f 6c 69  PsecON.TRC000oli   32: 50 18 80 00 1c 82 00 00 20 31 20 72 6f 6f 74 20  P....... 1 root   32: 50 18 80 00 b1 0a 00 00 12 02 00 06 02 00 00 31  P..............1   32: 50 10 80 00 92 31 00 00 -- -- -- -- -- -- -- --  P....1..........   32: 50 18 80 00 03 33 00 00 20 39 30 30 30 2f 38 30  P....3.. 9000/80   32: 50 18 80 00 31 20 00 00 42 00 00 07 03 80 00 49  P...1 ..B......I   32: 50 18 80 00 72 ee 00 00 46 00 00 05 03 80 00 49  P...r...F......I   32: 50 18 80 00 72 3a 00 00 46 00 00 05 03 80 00 49  P...r:..F......I   32: 50 10 80 00 d0 68 00 00 -- -- -- -- -- -- -- --  P....h..........   32: 50 18 80 00 33 87 00 00 3e 01 00 07 02 c0 00 48  P...3...>......H   32: 50 18 80 00 32 c3 00 00 3e 01 00 07 02 c0 00 48  P...2...>......H  720: 20 20 20 50 61 6c 6f 20 41 6c 74 6f 2c 20 43 41     Palo Alto, CA   32: 50 18 80 00 72 b8 00 00 46 01 00 07 02 c0 00 48  P...r...F......H   32: 50 18 80 00 72 91 00 00 46 01 00 05 02 c0 00 48  P...r...F......H   32: 50 10 80 00 bd 68 00 00 -- -- -- -- -- -- -- --  P....h..........   32: 50 18 80 00 6c e9 00 00 46 01 00 05 02 c0 00 48  P...l...F......H   32: 50 18 80 00 a6 c1 00 00 2d 72 77 2d 2d 2d 2d 2d  P.......-rw-----   32: 50 18 80 00 65 7a 00 00 46 00 00 05 03 80 00 49  P...ez..F......I   32: 50 10 80 00 b4 68 00 00 -- -- -- -- -- -- -- --  P....h..........   32: 50 18 80 00 62 31 00 00 46 15 00 05 03 80 00 49  P...b1..F......I root@hpeos004[.root] 

I tried as best I could but was unable to discern any pattern to the content of the packets.

30.4.8 Warnings regarding ICMP packets

Discarding or requiring ICMP messages (Internet Control Message Protocol messages, protocol value 1) to be encrypted or authenticated may cause connectivity problems. Normal network operation may require IP to exchange ICMP messages between end-to-end hosts and between an end host and an IP gateway (including router devices). IP may need to exchange ICMP packets with gateway nodes even though no user (end-to-end) services are being used to the gateways.

Be careful when configuring the default IPSec policy or IPSec policies that affect entire subnets because you may inadvertently cause ICMP messages to be discarded. You may also inadvertently require ICMP messages being transmitted or received from a non-IPSec gateway or router to be authenticated or encrypted, which will also cause ICMP packets to be discarded.

IP uses ICMP messages to transmit error and control information, such as in the following situations:

• IP may periodically send ICMP Echo messages to gateways to determine whether the gateway is up ( Gateway Probes ). If no response is received, the gateway is marked Dead in the IP routing table.

  This feature is controlled by the IP kernel parameter ip_ire_gw_probe . By default, this feature is enabled on all HP-UX systems.

• IP may use ICMP Echo messages with the Don't Fragment flag and ICMP Destination Unreachable messages with the Fragmentation Needed flag to set the Path Maximum Transmission Unit (Path MTU). This feature is controlled by the IP kernel parameter ip_pmtu_strategy . By default, HP-UX will attempt to send an ICMP Echo Don't Fragment message when initially establishing a route to a non-local destination, but no Echo Reply is required. However, you should ensure that the local system does not discard Fragmentation Needed packets that may be sent in response.

• IP may send ICMP Redirect messages to redirect traffic to a different gateway. The transmission of ICMP Redirect messages is controlled by the IP kernel parameter ip_send_redirects . By default, this feature is enabled on all HP-UX systems.

• IP may send ICMP Source Quench messages to request the source system to decrease its transmission rate. The transmission of ICMP Source Quench messages is controlled by the IP kernel parameter ip_send_source_quench . By default, this feature is enabled on all HP-UX systems.

Refer to the ndd manual page or recall our discussions on ndd in Chapter 15, "Basic IP Configuration" for information on checking or changing all these parameters' values.

30.4.8.1 CONCLUSIONS ON IPSEC

IPSec can provide privacy, authentication, authenticity, and effectively non-repudiation for packets transmitted at the IP level. This can be customized for relationships between individual hosts and customized for individual network services. While being computationally expensive (like any other encryption technology), IPSec can ensure that secure communication is possible even when using insecure applications such as telnet . While we have shown that even our telnet packets are now secure, I would still try to ensure that users stopped using these insecure applications in favor of applications such as ssh .