IPSec involves many component technologies and encryption methods. Yet IPSec’s operation can be broken down into five main steps. The five steps are summarized as follows:

Step 1: Defining Interesting Traffic

Determining what type of traffic is deemed interesting is part of formulating a security policy for use of a VPN. The policy is then implemented in the configuration interface for each particular IPSec peer. For example, in Cisco routers and PIX Firewalls, access lists are used to determine the traffic to encrypt. The access lists are assigned to a crypto policy such that permit statements indicate that the selected traffic must be encrypted, and deny statements can be used to indicate that the selected traffic must be sent unencrypted. With the Cisco Secure VPN Client, you use menu windows to select connections to be secured by IPSec. When interesting traffic is generated or transits the IPSec client, the client initiates the next step in the process, negotiating an IKE phase one exchange.

Step 2: IKE Phase One

The basic purpose of IKE phase one is to authenticate the IPSec peers and to set up a secure channel between the peers to enable IKE exchanges. IKE phase one performs the following functions:

• Authenticates and protects the identities of the IPSec peers
• Negotiates a matching IKE SA policy between peers to protect the IKE exchange
• Performs an authenticated Diffie-Hellman exchange with the end result of having matching shared secret keys
• Sets up a secure tunnel to negotiate IKE phase two parameters

IKE phase one occurs in two modes:

Main mode has three two-way exchanges between the initiator and receiver.

• First exchange -The algorithms and hashes used to secure the IKE communications are agreed upon in matching IKE SAs in each peer.
• Second exchange -This exchange uses a Diffie-Hellman exchange to generate shared secret keying material used to generate shared secret keys and to pass nonces, which are random numbers sent to the other party, signed, and returned to prove their identity.
• Third exchange -This exchange verifies the other side’s identity. The identity value is the IPSec peer’s IP address in encrypted form. The main outcome of main mode is matching IKE SAs between peers to provide a protected pipe for subsequent protected ISAKMP exchanges between the IKE peers. The IKE SA specifies values for the IKE exchange: the authentication method used, the encryption and hash algorithms, the Diffie-Hellman group used, the lifetime of the IKE SA in seconds or kilobytes, and the shared secret key values for the encryption algorithms. The IKE SA in each peer is bidirectional.

Aggressive Mode

In the aggressive mode, fewer exchanges are done and with fewer packets. In the first exchange, almost everything is squeezed into the proposed IKE SA values, the Diffie-Hellman public key, a nonce that the other party signs, and an identity packet, which can be used to verify the initiator’s identity through a third party. The receiver sends everything back that is needed to complete the exchange. The only thing left is for the initiator to confirm the exchange. The weakness of using the aggressive mode is that both sides have exchanged information before there is a secure channel. Therefore, it is possible to sniff the wire and discover who formed the new SA. However, aggressive mode is faster than main mode.

Step 3: IKE Phase Two

The purpose of IKE phase two is to negotiate IPSec SAs to set up the IPSec tunnel. IKE phase two performs the following functions:

• Negotiates IPSec SA parameters protected by an existing IKE SA
• Establishes IPSec security associations
• Periodically renegotiates IPSec SAs to ensure security
• Optionally performs an additional Diffie-Hellman exchange

IKE phase 2 has one mode, called quick mode. Quick mode occurs after IKE has established the secure tunnel in phase one. It negotiates a shared IPSec policy, derives shared secret keying material used for the IPSec security algorithms, and establishes IPSec SAs. Quick mode exchanges nonces that provide replay protection. The nonces are used to generate new shared secret key material and prevent replay attacks from generating bogus SAs.

Quick mode is also used to renegotiate a new IPSec SA when the IPSec SA lifetime expires. Base quick mode is used to refresh the keying material used to create the shared secret key based on the keying material derived from the Diffie-Hellman exchange in phase one.

Perfect Forward Secrecy

If perfect forward secrecy (PFS) is specified in the IPSec policy, a new Diffie-Hellman exchange is performed with each quick mode, providing keying material that has greater entropy (key material life) and thereby greater resistance to cryptographic attacks. Each Diffie-Hellman exchange requires large exponentiations, thereby increasing CPU use and exacting a performance cost.

Step 4: IPSec Encrypted Tunnel

After IKE phase two is complete and quick mode has established IPSec SAs, information is exchanged by an IPSec tunnel. Packets are encrypted and decrypted using the encryption specified in the IPSec SA.

Step 5: Tunnel Termination

IPSec SAs terminate through deletion or by timing out. An SA can time out when a specified number of seconds have elapsed or when a specified number of bytes have passed through the tunnel. When the SAs terminate, the keys are also discarded. When subsequent IPSec SAs are needed for a flow, IKE performs a new phase two and, if necessary, a new phase one negotiation. A successful negotiation results in new SAs and new keys. New SAs can be established before the existing SAs expire so that a given flow can continue uninterrupted.