IPSec, the Future of Network Security

Internet Protocol security (IPSec) uses cryptographic security services to protect communications over Internet Protocol (IP) networks. IPsec supports network-level peer authentication, data origin authentication, data integrity, data confidentiality (encryption), and replay protection. The Microsoft implementation of IPSec is based on Internet Engineering Task Force (IETF) standards.


Figure  IPsec contains a gateway and a tunnel in order to secure communications.

1. What Is IPSec?

Internet Protocol security (IPSec) is a framework of open standards for helping to ensure private, secure communications over Internet Protocol (IP) networks through the use of cryptographic security services. IPSec supports network-level data integrity, data confidentiality, data origin authentication, and replay protection. Because IPSec is integrated at the Internet layer (layer 3), it provides security for almost all protocols in the TCP/IP suite, and because IPSec is applied transparently to applications, there is no need to configure separate security for each application that uses TCP/IP.

IPSec helps provide defense-in-depth against:

  • Network-based attacks from untrusted computers, attacks that can result in the denial-of-service of applications, services, or the network
  • Data corruption
  • Data theft
  • User-credential theft
  • Administrative control of servers, other computers, and the network.

You can use IPSec to defend against network-based attacks through a combination of host-based IPSec packet filtering and the enforcement of trusted communications.

IPSec is integrated with the Windows Server 2003 operating system and it can use the Active Directory directory service as a trust model. You can use Group Policy to configure Active Directory domains, sites, and organizational units (OUs), and then assign IPSec policies as required to Group Policy objects (GPOs). In this way, IPSec policies can be implemented to meet the security requirements of many different types of organizations.

This section describes the solution that IPSec is intended to provide by providing information about core IPSec scenarios, IPSec dependencies, and related technologies.

The following figure shows an Active Directory-based IPSec policy being distributed to two IPSec peers and IPSec-protected communications being established between those two peers.

2.Two IPSec Peers Using Active Directory-based IPSec Policy

Figure : IPsec Two IPSec Peers Using Active Directory-based IPSec Policy.

The MicrosoftWindows implementation of IPSec is based on standards developed by the Internet Engineering Task Force (IETF) IPSec working group. For a list of relevant IPSec RFCs, see the “Related Information” section later in this subject.

3.IPSec Scenarios

IPSec is a general-purpose security technology that can be used to help secure network traffic in many scenarios. However, you must balance the need for security with the complexity of configuring IPSec policies. Additionally, due to a lack of suitable standards, IPSec is not appropriate for some types of connectivity. This section describes IPSec scenarios that are recommended, IPSec scenarios that are not recommended, and IPSec scenarios that require special consideration.

Recommended Scenarios for IPSec

IPSec is recommended for the following scenarios:

  • Packet filtering
  • End-to-end security between specific hosts
  • End-to-end traffic through a Microsoft Internet Security and Acceleration (ISA) Server-secured network address translator
  • Secure server
  • Layer Two Tunneling Protocol (L2TP) over IPSec (L2TP/IPSec) for remote access and site-to-site virtual private network (VPN) connections.
  • Site-to-site IPSec tunneling with non-Microsoft IPSec gateways.

3.1.Packet Filtering

Filtering Packets by Using IPSec

Figure : Filtering Packets by Using IPSec

As illustrated in this figure:

  • The internal network domain administrator can assign an Active Directory-based IPSec policy (a collection of security settings that determines IPSec behavior) to block all traffic from the perimeter network (also known as a demilitarized zone [DMZ], demilitarized zone, or screened subnet).
  • The perimeter network domain administrator can assign an Active Directory-based IPSec policy to block all traffic to the internal network.
  • The administrator of the computer running Microsoft SQL Server on the internal network can create an exception in the Active Directory-based IPSec policy to permit structured query language (SQL) protocol traffic to the Web application server on the perimeter network.
  • The administrator of the Web application server on the perimeter network can create an exception in the Active Directory-based policy to permit SQL traffic to the computer running SQL Server on the internal network.
  • The administrator of the Web application server on the perimeter network can also block all traffic from the Internet, except requests to TCP port 80 for the HyperText Transfer Protocol (HTTP) and TCP port 443 for HTTPS (HTTP over Secure Sockets Layer/Transport Layer Protocol [SSL/TLS]), which are used by Web services. This provides additional security for traffic allowed from the Internet in case the firewall was misconfigured or compromised by an attacker.
  • The domain administrator can block all traffic to the management computer, but allow traffic to the perimeter network.

3.2. End-to-End Security Between Specific Hosts

Figure : Securing Communications Between a Client and a Server by Using IPSec

The following figure shows domain controllers in two forests that are deployed on opposite sides of a firewall.

Figure : Securing Communications Between Two Domain Controllers in Different Forests by Using IPSec

3.3. End-to-End Traffic Through an ISA-Secured Network Address Translator

Figure : Securing Communications Through an ISA-Secured NAT by Using IPSec NAT-T

3.4. Secure Server

Figure :Securing an Application Server by Using IPSec

In this scenario, an application server in an internal corporate network must communicate with clients running Windows 2000 or Windows XP Professional; a Windows Internet Name Service (WINS) server, Domain Name System (DNS) server, and Dynamic Host Configuration Protocol (DHCP) server; Active Directory domain controllers; and a non-Microsoft data backup server. The users on the client computers are company employees who access the application server to view their personal payroll information and performance review scores. Because the traffic between the clients and the application server involves highly sensitive data, and because the server should only communicate with other domain members, the network administrator uses an IPSec policy that requires ESP encryption and communication only with trusted computers in the Active Directory domain.

3.5. L2TP/IPSec for Remote Access and Site-to-Site VPN Connections

You can use L2TP/IPSec for all VPN scenarios. This does not require the configuration and deployment of IPSec policies. Two common scenarios for L2TP/IPSec are securing communications between remote access clients and the corporate network across the Internet and securing communications between branch offices.


  • Windows IPSec supports both IPSec transport mode and tunnel mode. Although VPN connections are commonly referred to as “tunnels,” IPSec transport mode is used for L2TP/IPSec VPN connections. IPSec tunnel mode is most commonly used to help protect site-to-site traffic between networks, such as site-to-site networking through the Internet.
L2TP/IPSec for remote access connections

A common requirement for organizations is to secure communications between remote access clients and the corporate network across the Internet. Such a client might be a sales consultant who spends most of the time traveling, or an employee working from a home office. In the following figure, the remote gateway is a server that provides edge security for the corporate intranet. The remote client represents a roaming user who requires regular access to network resources and information. An ISP is used as an example to demonstrate the path of communication when the client uses an ISP to access the Internet. L2TP/IPSec provides a simple, efficient way to build a VPN tunnel and help protect the data across the Internet.

Figure : Securing Remote Access Clients by Using L2TP/IPSec

Establishing an L2TP/IPSec VPN Tunnel Between Sites

Figure : Establishing an L2TP/IPSec VPN Tunnel Between Sites

3.6. Site-to-Site IPSec Tunneling with Non-Microsoft Gateways

For interoperability with gateways or end systems that do not support L2TP/IPSec or Point-to-Point Tunneling Protocol (PPTP) VPN site-to-site connections, you can use IPSec in tunnel mode. When IPSec tunnel mode is used, the sending gateway encapsulates the entire IP datagram by creating a new IP packet that is then protected by one of the IPSec protocols. The following figure illustrates site-to-site IPSec tunneling.

Establishing an IPSec Gateway-to-Gateway Tunnel Between Sites

Figure : Establishing an IPSec Gateway-to-Gateway Tunnel Between Sites

4. Securing Communication Between Domain Members and their Domain Controllers

Using IPSec to help secure traffic between domain members (either clients or servers) and their domain controllers is not recommended because:

  • If domain members were to use IPSec-secured communication with domain controllers, increased latency might occur, causing authentication and the process of locating a domain controller to fail.
  • Complex IPSec policy configuration and management is required.
  • Increased load is placed on the domain controller CPU to maintain SAs with all domain members. Depending on the number of domain members in the domain controller’s domain, such a load might overburden the domain controller.

5. Securing All Traffic in a Network

In addition to reduced network performance, using IPSec to help secure all traffic in a network is not recommended because:

  • IPSec cannot secure multicast and broadcast traffic.
  • Traffic from real-time communications, applications that require Internet Control Message Protocol (ICMP), and peer-to-peer applications might be incompatible with IPSec.
  • Network management functions that must inspect the TCP, UDP, and other protocol headers are less effective, or cannot function at all, due to IPSec encapsulation or encryption of IP payloads.

6. Securing Traffic for Remote Access VPN Connections by Using IPSec Tunnel Mode

IPSec tunnel mode is not a recommended technology for remote access VPN connections, because there are no standard methods for user authentication, IP address assignment, and name server address assignment.

IPSec Uses That Require Special Considerations

The following scenarios merit special consideration, because they introduce an additional level of complexity for IPSec policy configuration and management:

  • Securing traffic over IEEE 802.11wireless networks
  • Securing traffic in home networking scenarios
  • Securing traffic in environments that use dynamic IP addresses

7. Securing Traffic Sent over 802.11 Networks

You can use IPSec transport mode to protect traffic sent over 802.11 wireless networks. However, IPSec is not the recommended solution for providing security for corporate 802.11 wireless LAN networks. Instead, it is recommended that you use either 802.11 Wired Equivalent Privacy (WEP) encryption or Wi-Fi Protected Access (WPA) and IEEE 802.1X authentication.

8. Securing Traffic in Home Networking Scenarios

  • In some cases, non-Microsoft VPN or firewall clients might disable the IPSec service, which is required for IPSec to function. If you encounter this problem, it is recommended that you contact the VPN or firewall vendor.

IPSec is not recommended for end users in general home networking scenarios for the following reasons:

  • The IPSec policy configuration user interface (IP Security Policy Management) is intended for professional network security administrators, rather than for end users. Improper policy configuration can result in blocked communications, and if problems occur, built-in support tools are not yet available to aid end users in troubleshooting.
  • Some home networking applications use broadcast and multicast traffic, for which IPSec cannot negotiate security.
  • Many home networking scenarios use a wide range of dynamic IP addresses.
  • Many home networking scenarios involve the use of a network address translator. To use IPSec across a NAT, both IPSec peers must support IPSec NAT-T.

9. Securing Traffic in Environments That Use Dynamic IP Addresses

IPSec depends on IP addresses for establishing secure connections, and it is often necessary for a server to have a static IP address in IPSec policy filters. In large network deployments and in some mobile user cases, using dynamic IP addresses at both ends of the connection can increase the complexity of IPSec policy design.

10. Active Directory

For organizations with large numbers of computers that must be managed in a consistent way, it is best to distribute IPSec policies by using Group Policy to configure Active Directory domains, sites, and organizational units (OUs), and then assigning IPSec policies as required to Group Policy objects (GPOs). Although you can assign local IPSec policies to computers that are not members of a trusted domain, distributing IPSec policies and managing IPSec policy configuration and trust relationships is much more time-consuming for computers that are not members of a trusted domain.

If you do use Active Directory-based IPSec policies, IPSec policy design and management must take into account the delays that result from the replication of Group Policy data from domain controllers to domain members. Often, the first step in troubleshooting a problem with IPSec connectivity is to determine whether the computer in question has the most current Group Policy assignment. To do this, you must be a member of the local Administrators group on the computer for which troubleshooting is being performed.