EVPN-VXLAN multi-fabric solution overview

This section provides a brief description of the single-hop single VXLAN fabric solution before discussing the multi-fabric solution in greater detail.

As shown below, the terms campus zone, DC, and site are used interchangeably depending on the environment.

Even though a single VTEP is shown in the diagrams below, VSX VTEPs for device redundancy should be utilized where applicable.

Single-hop single VXLAN fabric

Figure 1  Single-hop single VXLAN fabric across DCs/sites

The single-hop VXLAN solution with a single fabric provides the following:

  • Full mesh of VXLAN tunnels between VTEPs within a VXLAN fabric
  • A single VXLAN fabric with limited VTEP scale (128 VTEPs recommended)

EVPN-VXLAN multi-fabric with single fabric per site

Figure 2  Multi-hop VXLAN with multiple fabrics across DCs/sites

The EVPN-VXLAN multi-fabric solution with single fabric per site provides the following:

  • Full mesh of VXLAN tunnels between VTEPs within a VXLAN fabric
  • VTEPs between different fabrics avoid full mesh of tunnels and BGP peering between fabrics
  • Border VTEPs establish VXLAN tunnels between fabrics. A full mesh is required between border VTEPs.
  • Border VTEPs provide tunnel-to-tunnel forwarding (intermediate VTEP hopping) for L2/L3 traffic between fabrics and prevent loops between fabrics
  • Multiple VXLAN fabrics across sites provide improved VTEP scale

EVPN-VXLAN multi-fabric with multiple fabrics per site

Figure 3  EVPN-VXLAN multi-fabric with multiple fabrics per site - eBGP control plane

Figure 4  EVPN-VXLAN multi-fabric with multiple fabrics per site - VXLAN data plane

In environments that require multiple fabrics per site (e.g. multiple pods in a DC), the EVPN-VXLAN multi-fabric solution with multiple fabrics per site provides the following:

  • Full mesh of VXLAN tunnels between VTEPs within a VXLAN fabric
  • VTEPs between different fabrics avoid full mesh of tunnels and BGP peering between fabrics
  • The concept of a border leader VTEP is introduced (1 per site), this border leader VTEP will eBGP peer to all border VTEPs within the same site and also eBGP peer with border leaders in other sites as shown in Figure 3
    • 8325 can be deployed as border leader VTEP (e.g. Fabric1)
    • 8325 or 8360 can be deployed as border VTEP (e.g. Fabric11)
    • Any CX switch platform that supports EVPN-VXLAN can be deployed as VTEPs within a fabric
  • The border leader VTEP will also function as border VTEP for it's own fabric
  • Border leader and border VTEPs automatically establish VXLAN tunnels with other border leader and border VTEPs with the help of EVPN control plane. A full mesh of VXLAN tunnels is required between them as shown in Figure 4.
  • Border leader and border VTEPs provide tunnel-to-tunnel forwarding (intermediate VTEP hopping) for L2/L3 traffic between fabrics and prevent loops between fabrics
  • Multiple VXLAN fabrics provide improved VTEP scale to go beyond the limit of number VTEPs per fabric.

Related features

L2 VXLAN—selective split-horizon enforcement

Figure 5  L2 split horizon in VXLAN

Default split-horizon behavior in VXLAN switches

By default, VXLAN networks require full-mesh connectivity between VTEPs. Consider the full mesh single-fabric network shown in Figure 5 above. For example, Host H1 originates a BUM packet and one copy of this packet reaches Switch S2 and Switch S3 over the direct VXLAN tunnel from Switch S1 on the L2VNI corresponding to VLAN 10.  Now if Switch S2 and Switch S3 forward this packet on the same L2VNI to Switch S3 and Switch S2 respectively, a loop is formed in the overlay. Therefore, VTEPs perform split horizon and do not perform multi-fabric VXLAN bridging by default; VXLAN bridging between VXLAN tunnels is prohibited.

The full mesh assumption is valid when it comes multi-hop VXLAN bridging between intra-fabric and inter-fabric VXLAN tunnels. Here, split horizon enforcement should not be performed in order to deliver BUM traffic from one fabric to another. Therefore, the border VTEP has to enforce split horizon selectively as shown in Figure 6 below.

Figure 6  Selective split horizon in border VTEPs

Such selective split horizon can be enforced (in supported platforms) with additional configurations. This is achieved by associating by allowing traffic between intra-fabric and inter-fabric tunnels only. The switch enforces split horizon when VXLAN bridging between tunnels belonging to the same broadcast domain and disable split horizon enforcement across different broadcast domains. By associating all intra-fabric tunnels to one broadcast domain and all inter-fabric tunnels to another, the desired selective split horizon behavior is achieved.

Broadcast domain association is not per L2VNI, it is per VXLAN tunnel - the local and remote VTEP pair.

To change the default behavior, the command dyn-vxlan-tunnel-bridging-mode ibgp-ebgp is available to disable split horizon between iBGP learned VXLAN tunnels and eBGP learned VXLAN tunnels. This mandates the use of iBGP within a fabric and eBGP between fabrics.

EVPN route-map

The EVPN route map provides the following:

  • Provides EVPN route filtering support for VXLAN deployments
  • Allows a VTEP to apply an inbound or/and outbound route-map to a BGP peer under EVPN address-family context
    • Match conditions: IPv4/IPv6 address prefix-list, aspath-list, VNI (L2/L3)
    • Set clauses: local-preference, as-path prepend/exclude, IP next-hop
  • Required when EVPN-VXLAN multi-fabric is deployed to minimize the number of routes

Figure 7 shows an example with the objective and solution provided by the EVPN route map.

Figure 7  EVPN route map

Objective: Border VTEP2 should be preferred as outbound VTEP over Border VTEP1 for routes towards AS#65100

Solutions:

  • On VTEP2, set higher local-preference for EVPN routes learned from AS#65100 and lower local preference on VTEP1 for AS#65100 routes.
  • Set AS-path prepending on VTEP1 for EVPN routes learned from AS#65100.

EVPN next-hop-self

EVPN next-hop-self provides the ability to modify the next-hop VTEP IP address.

Figure 8 shows an example where VTEPs in AS#65001 use border leader VTEP (1.1.1.1/32) as the next-hop VTEP IP address to establish VXLAN tunnels since the expectation is for VTEPs within a fabric not to establish VXLAN tunnels with VTEPs in other fabrics.

Figure 8  EVPN next-hop-self