IPSEC Theory

Prior to reading this you should have a good understanding of the Internet Protocol (IP) and how it traverses networks locally and around the world. You will also need to be able to recall the structure of a packet (we’re leaving frame encapsulation out, as IPSec can function over more than just the layer 2 standard: Ethernet).

IPSec Theory & Application

Standard TCP/IP Packet Structure

What are VPNs?

Before we begin, we need to understand what a VPN is, as it is a primary function of IPSec. In its simplest form, it is merely a means to connect two remote sites together over a WAN.

Imagine that Site B (B) has been bought out by the company who owns site A (A), we need to join these sites at the network layer because B is going to need access to the servers in A which are only accessible over the private network and not the public network due to basic security policies set up by the company’s Information Officer to stop critical infrastructure from being publically accessible.

Ideally we could connect a direct fibre between these two sites, but imagine that this is not practical for the time being. What we need to do, then, is to logically join them rather than physically join them. This is achievable via a Virtual Private Network (VPN). In short, the VPN creates a virtual tunnel (link) between the two sites that acts much like a physical connection.

This tunnel links the two sites together so that LAN A can directly communicate with LAN B, because the router will know to route traffic over the tunnel when two sites are trying to communicate with one another.

What is IPSEC?

IPSec is an extension to IP which presents a layer of security  encapsulated in protocols from within its suite (e.g. ESP). Its primary goal is to provide:

• Confidentiality – This is achieved via encryption, which is the process of ciphering clear text into an unreadable format.
• Data Integrity – IPSec using HMAC (Hashed Message Authentication Code) to ensure that the data received was not altered during transit
• Authentication – This is the process of establishing a relationship with the peer you wish to send confidential/encrypted data to. The initial connection is achieved via pre-shared keys (PSK) or digital signatures, and continual authentication is controlled by HMAC.
• Antireplay Protection – This form of protection helps to stop attackers from repeating certain data to the peer (which would likely have changed the integrity of the packet in some way). This is achieved by sequencing the packets. When the peer receives the packet it can record the fact that it’s received the packet (e.g. #1) and when the attacker tries to replay packet #1, the peer will not accept it.

This protocol suite uses the following protocols to achieve the above points:

Authentication Headers (AH) – Provides, Data integrity, authentication and antireplay protection.

Encapsulating Security Payloads (ESP) – Provides confidentiality, authentication, integrity and antireplay protection via sequencing packets.

Security Associations (SA) – Provides the backbone algorithms used by AH and ESP. Examples are Internet Security Association and Key Management Protocol (ISAKMP), Pre Shared Keys (PSK), Internet Key Exchange (IKE, IKEv2), Kerberized Internet Negotiation of Keys (KINK) and IPSECKEY DNS records.

The Internet Key Exchange

The IPSec suite uses a framework provided by ISAKMP known as IKE to negotiate and establish secured VPN tunnels.

IKE works in two steps, or more specifically phases – in Phase 1 the VPN peers attempt to negotiate and authenticate with one another. In Phase 2 they negotiate keying materials and algorithms which will be used for data in transit over the VPN tunnel.

There is a newer version of IKE; IKEv2. This works similarly to its predecessor however it uses dynamic key exchange and peer authentication to simplify the key exchange flows, and it also fixes some vulnerabilities with IKEv1. Phase 1 is now known as IKE_SA, which has a message pair of IKE_SA_INIT used for the negotiation of tunnels. Phase 2 is known as CHILD_SA, with attributes as defined in the data policy.

Authentication Header

As its name represents, the function of AH is authentication. It does this with an additional header to the payload which is calculated based on the values within the datagram, but IPSec chooses which pieces to calculate upon depending on the IPSec mode, and the version of IP.

Similar to a frame’s CRC (checksum), AH is the value that verifies the authenticity of the datagram sent along the network, however rather than using a publically known algorithm to generate/verify the checksum, AH uses a privately (between two or more authenticated VPN peers) known algorithm. This is achieved via SAs which specify which algorithms and keys to use. The sender now uses this private algorithm to calculate the AH via the datagram, and the receiver ensures (s)he gets the same result (again, similar to CRC).

Encapsulating Security Payloads

This is an improvement upon AH, as it not only ensures integrity of the data but also protects against malicious users from examining the contents as well. ESP does this by encrypting the IP datagrams with a key to cipher it in a way only peers can decipher.

Unlike AH, it doesn’t have 1 header, rather it is split into 3 parts.

• ESP Header – Placed before the encrypted IP data, it is has two values: SPI & Sequence Number. The latter of which is used for anti-replay attacks.
• ESP Trailer – This is placed after the encrypted payload. It contains the padding required to align the data, and also contains the Next Header field.
• ESP Authentication Data – This fields contains only a Integrity Check Value (ICV) which is used for the function of integrity (similar to AH above).

IPSEC Modes

There are two modes of operation for IPSec: Transport mode and tunnel mode. Each mode affects what ESP or AH chooses in the IP datagram when protecting the headers. So in short, the tunnel mode describes how ESP/AH function.

Transport Mode

In this mode, the data that is protected is that within the transport layer (layer 4 OSI). When the transport layer packages data ready for transmission it sends it down to the IP layer (layer 3), which from IP’s perspective is its payload. That payload is the data that is protected with IPSec.

Transport Mode Frame

The TCP/UDP, Upper Layers and Application Data show the payload of IP that will be encrypted due to the transport mode proposed.

Transport mode is great for host-to-site VPNs, but doesn’t support NAT traversal and has a higher MSS.

Tunnel Mode

In this mode the entire IP datagram is encrypted, and IPSec adds its header on top of it, including a new IP Header. This masquerades all of the data (from IP address to application data) from those trying to peer through the tunnel. A great benefit of tunnel mode is that you can perform NAT Traversal, and it works well to get host-to-host VPNs.

The Steps to establish a VPN

In this step by step guide to IPSec, we are going to imagine that there are two routers trying to establish a site-to-site VPN tunnel.

Step 1: Establish Phase 1 Tunnel via IKEv1

Our first step is to establish the phase 1 tunnel, which will be used by the two routers to communicate VPN management traffic (e.g. keepalives to keep the tunnel open), and will not be used to send user data. In IKEv1 there are two modes that we can choose from: main mode or aggressive mode, the former is our most secure option and usually the default option, so we’ll stick with this.

When a router receives traffic that needs to be encrypted and there is no Phase 1 IKE tunnel, this router becomes the initiator of phase 1 and sends its configured parameters to the peer router which originated the data. There are 5 parameters that are imperative to establish the tunnel.

• The Hash Algorithm – Examples are MD5 or SHA-1
• Encryption Algorithm – In order of weakest to strongest: DES, 3DES or AES (the latter contains various key lengths, where the higher the number, the more secure the algorithm).
• Diffie-Hellman (DH) group – The DH Group refers to the length of the key used for the DH key exchange. DH is used to generate symmetric keys that can be used by the two VPN peers for the symmetrical algorithm chosen above – However the process of DH is actually Asymmetric (uses Public & Private key pairs) and is typically CPU intensive
• Authentication Method – In order to verify the peers on the other end of one another, we need to authenticate them with either a Pre-Shared key or a digital signature. The pre-shared key is only used for establishing the phase 1 connection, and authenticating the peer – this is not used for encrypting user data (this is instead controlled by DH).
• Lifetime – This is how long until the Phase 1 tunnel is kept before being torn down (and then re-established). This parameter doesn’t need to match on both ends, as IKE will chose the shortest time because it’s the most secure method (when the tunnel is re-established the keys generated in DH are changed so an attacker has less time to crack it). Also note that the default time is 1 day.

Note that these parameters are sent in plain-text but hold no confidential data such as the keys, only the parameters to set up the algorithms.

Step 2: DH Key Exchange

Now that we have established our tunnel, we can begin generating the symmetric key that will be used for encrypting our user’s data. This key’s size is dependent on the DH group that you choose:

• DH Group 1 – 768 bit modulus
• DH Group 2 – 1024 bit modulus
• DH Group 5 – 1536 bit modulus
• DH Group 14 – 2048 bit modulus
• DH Group 19 – 256bit elliptic curve
• DH Group 20 – 384 bit elliptic curve – NGE (Next Generation Encryption)
• DH Group 21 – 512 bit elliptic curve – NGE
• DH Group 24 – modular exponentiation group with 2048 bit modulus and 256 bit prime order subgroup – NGE

The above list shows what each group correlates to, where groups 1-5 should be avoided as they have either been cracked already or are on their way to being insecure, group 14 & group 19 are acceptable, but should be upgraded in the future, and groups 20, 21 & 24 are next generation groups which are expected to be secure and scalable for the next 2 decades.

Step 3: Authenticate THEN Build IPSec Tunnel

Now we have our key, we can begin to open up a second tunnel known as the IPSec tunnel (achieved in Phase 2), which will be used to send user data between sites securely. Before we can start the phase 2 tunnel, we need to authenticate the peer with the authentication method chosen in step one (PSK or RSA Digital Signatures), if this matches then phase 1 has been completed.

In order to build the second tunnel, the routers use the phase 1 tunnel to send management information for forming the phase 2 tunnel. With this information, a transform set is compared and contains details of the hashing and encryption algorithm These are not the same details as  we specified in step 1. That sounds confusing, but step 1 set up our IKE Policy which is used for securing the phase 1 tunnel, whilst we need additional information to secure the phase 2 tunnel, which requires the Hashing algorithm and the encryption algorithm.

Once this has been negotiated, the phase 2 tunnel opens up ready for sending encrypted traffic between sites for the users. From the perspective of the WAN (if you sniffed the packets) you would only ever see the two routers talking to one another with an encrypted payload and wouldn’t know what else was happening (e.g. which peer is speaking to which).

Summary

IPSec is a wonderful tool for keeping your data safe from malicious users who attempt to intercept traffic between two sites, so long as you se tup your VPN with the best possible settings. It has been around for decades and will be around for much longer due to its wonderful scalability.