Secure Data Center Interconnect
Table of contents
Overview
Organizations implement multiple data centers for a variety of reasons. These may include:
- Provide active redundancy of business-critical applications,
- Provide geographically distant backup for organizational data,
- Accommodate geo-based service delivery,
- Extend the data center to public cloud, private cloud, or colocation infrastructure.
Organizations also may have multiple data centers following mergers or acquisitions.
Data transport between remote data center locations is susceptible to eavesdropping and loss of integrity. Data centers can share data over the public Internet, commercial Layer 2 or Layer 3 transport services, cloud-provider transport services, and leased fiber- or privately-owned fiber plants.
When using the Internet, commercial, or cloud services, data in motion are visible to third parties during transmission. Privately-owned and leased fiber infrastructure are much more secure, but are susceptible to tapping at signal regeneration and termination points.
The risks of losing private data include financial loss, legal action, loss of intellectual property, and loss of reputation.
IPsec tunnels are the industry-standard method for securing data in motion over untrusted infrastructure or third-party networks, including communication between data centers.
Network Address Translation (NAT) services often are combined with IPsec services, when there is a need to accommodate overlapping IP address space. This need often arises when connecting data centers from previously independent organizations following mergers and acquisitions. Use of NAT services is also common in multi-tenant environments such as private cloud and colocation environments.
IPsec
IPsec mitigates the risks associated with transporting data between data center locations by providing:
- Data confidentiality between locations via payload encryption,
- Authentication to verify that both parties in the conversation are known and trusted,
- Data integrity checks to detect and prevent data alteration during transmission.
The simplified diagram below illustrates IPsec Encapsulating Security Payload (ESP) tunnel encryption of a standard IP packet into an IPsec ESP tunnel packet.
The HPE Aruba Networking CX 10000 switch implements the IPsec ESP tunnel protocol in hardware using the on-board AMD Pensando DPU. The CX 10000 also provides NAT services, often combined with IPsec services, when connecting data center locations that must accommodate overlapping IP address space.
Use Cases
Traditional Two-Tier data center topologies, EVPN-VXLAN overlay topologies, and edge data centers can use IPsec to secure communication between locations.
Site-to-Site Transport
When transmitting data using commercial transport services such as metro-Ethernet or MPLS, there is a risk that data can be observed by a third party. Using dedicated fiber substantially reduces risk, but data in motion can still be observed at signal regeneration and termination points outside an organization’s direct administrative control, such as at a colocation facility.
The diagram below illustrates the concept of linking two data centers using an IPsec tunnel to protect data. IPsec high-availability and routing concepts are detailed later in this document.
Note: IPsec is a vendor-agnostic suite of protocols enabling authentication and encryption between peers in a multi-vendor environment. Deploying CX 10000 switches on both sides of an IPsec ESP tunnel is recommended for effective, efficient support and troubleshooting.
Internet Transport
Establishing connectivity among data centers using commodity Internet links provides economic benefits, but at a much higher risk of data eavesdropping and corruption, making IPsec a critical tool for securing data. IPsec tunneling is a common approach for connecting to public cloud, private cloud, and colocation data centers. SD-WAN appliances with multiple Internet uplinks typically are positioned in the data path to improve underlying network resiliency.
The diagram below illustrates the concept of linking two data centers using an IPsec tunnel over commodity Internet.
Dedicated Public Cloud Transport
Infrastructure as a Service (Iaas) public cloud providers offer direct connections into virtual cloud infrastructure. Examples include AWS Direct Connect, Microsoft’s Azure ExpressRoute, and Google Cloud’s Cloud Interconnect services. In each case, data transmission occurs over infrastructure outside the administrative control of an organization.
IPsec tunnels can be terminated on VPN services supplied by the public cloud provider, or on a more advanced VPN appliance provisioned in the virtual private cloud environment.
CX 10000 IPsec Overview
CX 10000 switches running AOS-CX 10.12.1000 and higher support hardware-accelerated IPsec services. AOS-CX 10.15 adds additional IPsec capabilities, including support of floating static routes to establish backup tunnels and dynamically learned BGP routes.
Each CX 10000 has two AMD Pensando DPUs. IPsec tunnel encryption and decryption are performed on the DPUs. Each DPU supports 135 Gbps full-duplex IPsec operation, for a total of 270 Gbps IPsec performance on a single CX 10000 switch.
When using an EVPN-VXLAN overlay topology, IPsec tunnels are configured on border leaf switches. When using a Two-Tier topology, IPsec tunnels are configured on the data center core switches. In most cases, the switch profile should be configured as l3-core.
Note: A CX 10000 terminating IPsec tunnels requires assigning the l3-core or spine switch profile.
An IP address assigned to a VLAN SVI in a non-default VRF must be configured as the IPsec tunnel source. IPsec termination is not supported in the default VRF.
Traffic destined to an IPsec tunnel on a CX 10000 must be received in the same VRF context in which the tunnel is configured.
When establishing a tunnel, the CX 10000 supports passphrase and certificate-based authentication. For certificate-based authentication, the certificate authority (CA) used to generate the certificate of the remote IPsec host must be installed on the CX 10000.
A CX 10000’s physical interfaces are assigned port personas that govern operational behavior for traffic processed by the DPU. When using IPsec, ports facing external networks are assigned the uplink persona. Ports facing data center switches and hosts are assigned the access persona, which includes spine links in a spine-and-leaf network.
EVPN-VXLAN
When using IPsec in an EVPN-VXLAN data center, the border leaf is dedicated to that function. Extending Layer 2 VNIs and directly connecting hosts to the CX 10000 border leaf terminating IPsec tunnels is not supported.
Inter-VRF route leaking (IVRL) is not supported on the border leaf terminating IPsec tunnels. Each overlay VRF requiring IPsec services must have its own IPsec tunnel configuration. When connecting discrete EVPN-VXLAN fabrics using IPsec, data center connectivity is typically extended on a per-VRF basis using a VRF-lite strategy.
Note: An IPsec services VRF can be created to encrypt host traffic for multiple overlay VRFs by performing IVRL on a services leaf or standard leaf in the fabric that does not terminate IPsec tunnels.
Extending Layer 2 and Layer 3 overlays to establish a multifabric topology is not supported, because the default VRF that carries VXLAN encapsulated traffic does not support IPsec operation. The IPsec tunnel does not support extending Layer 2 reachability between fabrics.
Two-Tier
Many traditional Two-Tier data centers use the default VRF for data center host traffic. When using CX 10000 IPsec services, a non-default VRF must be configured for data center connectivity.
IVRL is not supported, when using IPsec in a Two-Tier data center. When multiple VRFs are present, each VRF can be extended to a remote data center by establishing an IPsec tunnel for each VRF using a VRF-lite strategy.
Edge Date Center
Hosts can be directly connected to a CX 10000 edge data center using IPsec to secure communication to a remote data center. All hosts must be connected to a non-default VRF. IVRL is not supported, but multiple VRFs can be extended by configuring an IPsec tunnel per VRF using a VRF-lite strategy.
The diagram below illustrates connecting an edge data center to a traditional Two-Tier data center use redundant IPsec tunnels. 
Routing
Traffic routed through an IPsec tunnel is identified using two methods: static routes and BGP peerings. IPsec tunnels do not support Layer 2 bridging traffic to extend broadcast domains or VLANs between locations.
Static routes manually specify traffic by assigning a prefix with a tunnel identifier as the next-hop. As of AOS-CX 10.15, a backup IPsec tunnel is supported using a floating static route, where an assigned administrative distance to the route determines the preferred IPsec tunnel. When the tunnel associated with the preferred route fails, that route is withdrawn, and the backup route is installed into the forwarding table, instructing the switch to send traffic down the backup tunnel.
When using static routes in an EVPN-VXLAN network, the routes are redistributed into the IPv4 or IPv6 BGP address family for the same VRF context as the tunnel. Other VTEPs in the same fabric learn the redistributed route from an EVPN Type 5 advertisement originated by the border leaf.
A Two-Tier data center can use either OSPF or BGP as the routing protocol to external networks. Static routes should be redistributed into the appropriate VRF context of the routing protocol used to establish connectivity outside the data center.
When using AOS-CX 10.15 or later, BGP peerings between IPsec tunnel endpoints may be used to dynamically exchange IP prefixes that should be forwarded in the IPsec tunnel instead of static routes. The CX 10000 supports up to 1,024 learned prefixes per VRF for IPsec redirection. BGP peerings are supported when using standalone or active/active high-availability tunnels.
When using EVPN-VXLAN overlays, host routes should be filtered from the BGP exchange over IPsec. Additionally, the learned prefixes are propagated dynamically to all VTEPs in a fabric as EVPN Type 5 advertisements, without requiring additional redistribution.
When using a traditional Two-Tier topology, redistribution is not required when BGP is used to connect to external networks. When using OSPF, BGP routes must be redistributed into the OSPF instance running in the corresponding VRF context.
When using a VSX pair and route redistribution, both switch members must perform redistribution. If the VSX ISL fails, only the primary VSX member performs redistribution.
When configuring BGP adjacencies and redistribution, route maps are recommended to reduce the risk of errors and unintended network disruption.
IPsec Policy
An IPsec policy is created in Pensando Policy and Services Manager (PSM). The policy defines authentication, encryption, and high-availability parameters for a tunnel, known as a security association (SA). Both local and remote peers of the IPsec tunnel must be configured with parameters such that they can mutually agree on the Internet Key Exchange (IKE) version, an authentication method, and an encryption algorithm for the SA.
IPsec policy is combined with firewall policy, and optional NAT policy. If an explicit firewall policy is not defined, an implicit firewall policy permits all traffic. The diagram below illustrates the processing order of policies, based on the direction of traffic:
PSM Policy Distribution Targets (PDTs) provide the flexibility to apply firewall, NAT, and IPsec policy to a subset of CX 10000 switches. PDTs are defined in PSM to scope the application of IPsec related policies to the appropriate set of CX 10000 switches.
HPE Aruba Networking Fabric Composer is recommended for configuring PSM policy.
High Availability
In addition to standalone IPsec tunnels, the CX 10000 supports two high-availability modes when using VSX pairs:
- Active/Active
- Active/Passive
Active/Active
Active/active redundancy is the recommended high-availability strategy, where each CX 10000 in a VSX pair establishes an active tunnel to a remote site. It is recommended that the remote site uses a complementary pair of CX 10000 switches. However, the remote termination can be on a single physical or virtual device.
Active/active redundancy provides fast convergence in case of failure, since there is no need to initiate a new IPsec SA before forwarding traffic.
VSX supports flow synchronization between member pairs to share state and support asymmetric traffic forwarding. In case of switch failure, the other VSX member can take over established flows seamlessly. VSX active forwarding enables either member switch to forward decrypted IPsec traffic directly toward downstream data center hosts without consuming the ISL between the VSX pair.
The following diagram illustrates active/active IPsec tunnels between CX 10000 switches at two data center locations.
The diagram below illustrates active/active IPsec tunnels between CX 10000 switches and a single remote device that supports active/active tunnels. This is common when terminating tunnels to a third-party host, such as a virtual appliance in the public cloud or a physical firewall appliance at a branch data center.
When IPsec tunnels terminate remotely on a single physical device, the IPsec appliance typically supports physical redundancy using a second appliance in passive mode that can become active if the primary appliance fails. Dual tunnels provide tunnel forwarding diversity for a single device, while physical diversity is achieved using a standby physical device.
Active/Passive
Active/passive high-availability is used when connecting to a third-party device that supports only a single active tunnel. This design has higher convergence times than an active/active design following a network failure, and it should be used only when necessary.
Active/passive operation requires Virtual Router Redundancy Protocol (VRRP) on the CX 10000 VSX pair. The active IPsec tunnel follows ownership of the VRRP Virtual IP (VIP) between the pair. The standby tunnel is associated with the VRRP standby interface. When the VRRP standby interface becomes the active VRRP VIP owner, it establishes the active IPsec tunnel. When using an MC-LAG between the VSX pair and the upstream WAN device, VRRP dual-active forwarding enables either VSX member to route received traffic destined for the VRRP VIP’s MAC address.
The diagram below illustrates an active/passive IPsec tunnel topology.
Using this strategy, it is common for outbound data center traffic to land on the CX 10000 switch that does not own the active IPsec tunnel. VSX redirects the traffic through the ISL to the CX 10000 with the active IPsec tunnel.
MTU
When IPSec tunnels connect over the Internet, the default Layer 3 IPsec interface MTU of 1500 is recommended. This value sets the TCP maximum segment size (MSS) to 1403 bytes to accommodate the 97-byte overhead of the IPsec headers, which avoids fragmentation of TCP-based traffic. Fragmentation may result in data loss.
When data center interconnections can support a higher MTU value end-to-end, such as when using leased fiber or a metro-Ethernet provider, it is recommended to set the Layer 2 and Layer 3 MTUs to the highest value supported across all devices.
Caveats
Consider the following caveats when using CX 10000 IPsec tunnels as of AOS-CX 10.15. The AOS-CX 10.15 documentation contains a complete list of caveats.
- Multicast is not supported.
- IPv6 addressing is not supported for tunnel endpoints.
- ACLs cannot be applied to IPsec tunnel interfaces.
- Routes exceeding 1024 IPsec routes per VRF are dropped. Ordering preferred routes cannot be guaranteed if more routes are learned than can be supported.
- A floating static route is not supported as a backup path for the same prefixes learned via BGP over an IPsec tunnel.
- BFD for BGP over an IPsec tunnel is not supported.
- Route-based ECMP does not support multiple IPsec tunnel next-hops.
Management
HPE Aruba Networking Fabric Composer provides single-pane-of-glass management of IPsec tunnels, as part of its NetOps and SecOps capabilities. Both AOS-CX and PSM components can be configured and monitored using Fabric Composer, including VRFs, interfaces, static routes, BGP peering, IPsec policy, NAT policy, and firewall policy. Fabric Composer safely enables collaborative management across multiple teams by supporting customizable permission assignments using Role-Based Access Control (RBAC).