This blog dives into the top network security protocols that safeguard enterprise networks. It explores IPsec, TLS, and WPA3/802.1X, breaking down how they work and why they matter. Plus, you’ll get practical tips for identifying which security protocols protect your network.
What network security protocols do
The purpose of network security protocols is to protect the traffic that is traversing the network. The ways traffic is protected are:
- Authentication of the network nodes before sending traffic between the network nodes to ensure only authorized network nodes can access the network and send data.
- Encryption of the traffic between network nodes to protect the confidentiality of the data sent over the network.
- Message integrity and authentication checks to ensure the traffic has not been tampered with and comes from a legitimate sender.
Core network security protocols
Below are the leading network security protocols you should be familiar with. Knowing that these protocols perform authentication, encryption, and message integrity is a fundamental first step in understanding network security protocols.
- The Wi-Fi Alliance defines how to protect traffic over Wi-Fi networks in their Wi-Fi Protected Access (WPA2/WPA3) specifications. WPA2/WPA3 Enterprise mandates IEEE 802.1X Network Access Control. 802.1X leverages the Extensible Authentication Protocol (EAP) to mutually authenticate the wireless device and wireless network before sending traffic and to establish keys for subsequent encryption and message integrity.
- IETF Internet Protocol Security (IPsec) defines a suite of protocols. The Internet Security Association and Key Management Protocol (ISAKMP) mutually authenticate two IP network nodes, such as a centrally located edge IP router and a remote branch IP router. Once authenticated, ISAKMP establishes the keys for subsequent traffic encryption and message integrity checks. The Encapsulating Security Payload (ESP) protocol provides encryption and message integrity, while the Authentication Header (AH) provides message integrity only.
- IETF Transport Layer Security (TLS) defines multiple protocols. The TLS Handshake Protocol enables mutual authentication between a client and a network server. Once authenticated, the Handshake Protocol establishes the keys for subsequent traffic encryption and message integrity. The Application Data Protocol carries the encrypted data. The TLS Record Protocol encapsulates either the Handshake Protocol or Application Data and carries the message integrity hash (see Figure 1).
Protecting different network layers
Network security protocols work at different layers of the network. 802.11/802.1X operates at the link layer, that is, between two physical nodes. IPsec operates at the network layer between two network nodes with IP addresses. TLS operates at the session layer between a client and a server. Figure 2 shows a scenario where a user is connecting to a wireless network, is using a VPN service protected with IPsec, and is browsing the web. In this scenario, device authentication, encryption, and message integrity are done at all three layers!
Mapping the security protocols in your network
Creating a map of the network security protocols used in your network is essential. Ideally, this map should show which network security protocols are used between which network nodes. Both physical and cloud-based networks should be included. The benefits of doing this include:
- Improves security & threat management by identifying the parts of the network that are protected and those that are unprotected and may be vulnerable to attack. The organization’s security policies should define the rules for protecting network traffic. Not all network traffic must be protected, such as traffic between routers in the same physical closet or data center. Creating a map of network security protocols helps ensure the correct implementation of these policies.
- Facilitates compliance with regulatory requirements for organizations that need to monitor and secure network traffic. Regulations may include GDPR, HIPAA, and PCI-DSS.
- Assists with network optimization & performance. Applying security measures impacts network performance. Identifying active protocols can enable you to check if network resources are wasted on redundant security measures, such as encrypting data at multiple layers.
Pro tips to identify network security protocols in use
Determining which network security protocols are in use in your network is not easy. Here are some Pro Tips to get you started.
Pro tips for discovering IEEE 802.11/802.1X wireless networks:
- Look at the wireless controller configuration to determine whether it deployed WPA2 or WPA3 . When deploying WPA2/WPA3 Enterprise, the controller is configured with the address of the AAA server (usually a RADIUS server) used for authentication. The controller configuration also shows the cipher suite being used, namely CCMP/AES or GCMP/AES.
Pro tips for finding IETF IPsec-based networks, look at:
- The configuration of the network connections between edge border routers/gateways that link central on remote branch locations. Typically, digital certificates are used for authentication, but other supported methods include Pre-Shared Keys (PSK) or an EAP authentication method. Also, check what encryption and message integrity hashing algorithms are in use.
- VPN services between end-user devices and a VPN server. Although IPsec is common for VPNs, TLS-based VPNs are increasingly popular.
IETF TLS has many uses within enterprise networks, including protecting traffic to/from email servers and file transfers (FTPS). Pro tips for looking at two crucial TLS usages in your network are:
- Protection of web server traffic, specifically through Hypertext Transfer Protocol Secure (HTTPS). Do a Wireshark packet capture to observe the traffic on the web server. Due to the high volume of traffic seen on most web servers, it is advisable to use a Wireshark capture filter that only selects traffic from HTTPS port 443 and TLS handshake messages. Figure 3 shows a Wireshark packet capture filtering on TLS Server Hello. Traffic is protected by TLS 1.2 and TLS 1.3. TLS 1.3 provides improved security. The TLS Server Hello also shows the cryptographic suite (encryption and message integrity algorithms) to protect subsequent application data.
- Protection of communications and API calls to cloud platforms such as AWS, Google Cloud, and Microsoft Azure. Cloud resources are typically accessed through web protocols such as HTTPS. There are two areas worth examining. First, does the cloud provider’s policy regarding support for legacy versions of TLS? For example, Microsoft will discontinue support for the cryptographically weaker TLS 1.0 and 1.1 to the Azure Application Gateway at the end of August 2025 . Second, whether your organization is leveraging the cloud provider’s firewall manager and whether there is a TLS firewall policy.
Conclusion and call to action
Network security protocols are complicated and laden with acronyms. By focusing on the key functions every protocol performs, you can start to build a framework for understanding these protocols and how they protect the network.
After reading this blog, you should:
- Describe why knowing the security protocols used in your network matters.
- Create a plan to identify security protocols in your network.
- Identify other network security protocols that may be operational in your network, such as Secure Shell (SSH).
- Obtain permission to look at configuration files and perform any packet captures.
- Start identifying the network security protocols used in your network.
