EVPN multi-homing support for L3 services

1.1. Problems with unicast load-balancing from core to CE

The layer-2 hashing performed by CE over its LAG port means that only one service PE may populate its ARP/ND cache. In Figure 1, if CE1 ARP/ND responses always hash to PE1, then PE2's ARP/ND table remains empty. Traffic from remote PEs can be received by either service PE. Traffic that reaches PE2 does not find an ARP entry and is dropped.¶

Solution: Synchronize ARP/ND entries using EVPN RT-2 routes as described in Section 3.6.¶

1.2. Problems with multicast from core to CE

Multicast IGMP/MLD join messages from CE may always hash to a single PE due to LAG hashing behavior. When PIM runs on both redundant PEs, PIM hello messages from each PE are not visible to the other PE because the CE cannot switch traffic between LAG members. Both PEs become PIM Designated Router (DR). However, IGMP joins for a given multicast group may hash to only one PE, so only that PE programs the multicast route and sends PIM joins.¶

Solution: Synchronize IGMP/MLD state using EVPN RT-7/RT-8 routes as described in Section 3.7.¶

1.3. Problems with IGP adjacencies over the LAG port

A layer-3 CE device connecting to redundant PEs may establish an IGP adjacency on the bundle port. The adjacency forms to only one PE, so IGP customer routes are only present on that PE. This prevents load-balancing benefits as only one PE advertises customer routes to the core.¶

                  <---------+
                            | IGP Adj
    +-------+               |
    |       | 192.0.2.1/24  |
    | PE1   +-----------+   |
    |       |           |   |
    |       |           |   +
    +-------+           |
                        |
        +               |  +------+
  RT5   |             L |  | CE1  +------>H1
  Sync  |             A +->+      |
        v             G |  |      |
                        |  |      +------>R1
    +-------+           |  +------+
    |       |           |    192.0.2.2/24
    | PE2   +-----------+
    |       | 192.0.2.1/24
    |       |
    +-------+

Figure 2 provides an example of this use case, where CE1 forms an IGP adjacency with PE1 (example: ISIS or OSPF), and advertises its H1 and R1 routes into the IP-VRF of PE1. PE1 may then redistribute this IGP route into the core as an L3 service. Any remote PEs are only aware of the service from PE1, and cannot load balance through PE2 as well.¶

Solution: Synchronize IGP learned routes using EVPN RT-5 routes as described in Section 3.8.¶

Note: BGP PE-CE protocols require further study.¶

1.4. Problems with supporting multiple subnets on same ES in all active mode

When a CE supports multiple subnets using VLANs over a single LAG interface, each VLAN maps to a separate L3 sub-interface on the PE. When the PE synchronizes host reachability using EVPN RT-2 routes, standard RT-2 advertisements do not indicate which sub-interface (VLAN) the host belongs to. The peering PE cannot determine the correct destination sub-interface when multiple sub-interfaces share the same ESI.¶

The same problem occurs with IGMP/MLD route synchronization using RT-7 and RT-8.¶

Solution: Use the Ethernet Tag-ID field to carry the VLAN ID in all route sync messages (RT-2, RT-5, RT-7, RT-8) to identify the specific sub-interface.¶

Note: This document focuses on L3 sub-interfaces. Mixed L2/L3 sub-interfaces require further study.¶

1.5. Acronyms

BD:

Broadcast Domain¶

BE:

Bundle Ethernet Interface aka. L3 LAG interface¶

DF:

Designated Forwarder¶

DR:

Multicast Designated Router¶

EC:

BGP Extended Community¶

ES:

Ethernet Segment. When a customer site (device or network) is connected to one or more PEs via a set of Ethernet links, then that set of links is referred to as an 'Ethernet Segment'.¶

ESI:

Ethernet Segment Identifier. A unique non-zero identifier that identifies an Ethernet Segment is called an 'Ethernet Segment Identifier'.¶

ES-LAG:

This refers to multi-homing scenario where peering PEs, connected to same CE, are two, three or more.¶

ETAG:

Ethernet Tag. An Ethernet tag identifies a particular broadcast domain, e.g., a VLAN. An EVPN instance consists of one or more broadcast domains.¶

EVI:

An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN. It is used to assist a L3 VRF for route synchronization.¶

GRT:

Global Routing Table¶

ICL:

Inter Chassis Link¶

IGMP:

Internet Group Management Protocol¶

IGP:

Interior Gateway Protocol¶

IP-VRF:

A VPN Routing and Forwarding table for IP routes on an PE. The IP routes could be populated by EVPN and IP-VPN address families. An IP-VRF is also an instantiation of a layer 3 VPN in an PE.¶

MAC-VRF:

A Virtual Routing and Forwarding table for Media Access Control (MAC) addresses on a PE. A MAC-VRF is also an instantiation of an EVI in a PE¶

MC-LAG:

Multi-Chassis Link Aggregation Group (MC-LAG).¶

MLD:

Multicast Listener Discovery.¶

PE:

Provider Edge.¶

PIM:

Protocol Independent Multicast.¶

RD:

Route Distinguisher used in BGP.¶

RP:

Multicast Rendezvous Point.¶

RT:

Route-Targets used in BGP¶

RT-2:

EVPN route type 2, i.e., MAC/IP advertisement route, as defined in [RFC7432].¶

RT-5:

EVPN route type 5, i.e., IP Prefix route, as defined in Section 3 of [RFC9136].¶

RT-7:

EVPN route type 7, i.e., Multicast Join Synch Route, as defined in Section 9.2 of [RFC9251].¶

RT-8:

EVPN route type 8, i.e., Multicast Leave Synch Route, as defined in Section 9.3 of [RFC9251].¶

1.6. Requirements

The multi-homing solution described in this document satisfies the following requirements:¶

1. MUST support Layer-3 access interfaces¶

2. MUST support Layer-3 access sub-interfaces¶

3. MUST support unicast and multicast VPN services¶

4. SHOULD support IGP route synchronization¶

5. SHOULD support global routing table (GRT) services¶

6. MUST support all-active load-balancing mode¶

7. MAY support single-active load-balancing mode¶

8. MUST support port-active load-balancing mode¶

9. SHOULD avoid Layer 2 constructs (EVI, MAC-VRF, BD, IRB) for L3 state synchronization¶

3.1. Example Topology

Consider the Figure 3 topology, where two AC-aware bundling interfaces are configured. Interface BE1 on PE1 and PE2 shares a LAG with switch SW1 and supports two customer VRFs with overlapping subnets on VLAN 1 and VLAN 2. Interface BE2 supports a single customer VRF on native VLAN.¶

+------
|     +-------+ BE1.1 (192.0.2.1/24)
| PE1 || BE1  +---------------------------------+
|     || ESI-1|                                 |
|     ||      | BE1.2 (192.0.2.2/24)            |
|     ||      +-------------------------+       |
|     +-------+                         |       |
|     |                                 |       |
|     +-------+ BE2 (198.51.100.1/24)   |       |
|     || BE2  +------------------+      |       |
|     || ESI-2|                  |      |       |
|     ||      |                 +v----+ |       |
|     ||      |                 |CE1  | |       |
|     +-------+                 |.2   | |       |
+------                         |CUST1| |       |
                                +^----+ |       |
+------                           |     +v-----+-v----+
|     +-------+                   |     |SW1   |      +-->H1(.2)
| PE2 || BE2  +-----<-------------+     |CUST2 |CUST1 |
|     || ESI-2| BE2 (198.51.100.1/24)   +^-----+-^----+
|     ||      |                         |       |
|     ||      |                         |       |
|     +-------+                         |       |
|     |                                 |       |
|     +-------+ BE1.2 (192.0.2.2/24)    |       |
|     || BE1  +-------------------------+       |
|     || ESI-1|                                 |
|     ||      | BE1.1 (192.0.2.1/24)            |
|     ||      +---------------------------------+
|     +-------+
+------

PE(1,2):
CUST1-VRF (IP-VRF1)
CUST2-VRF (IP-VRF2)

SW1:
CUST1-Subnet1: (192.0.2.1/24) (VLAN 1)
CUST2-Subnet1: (192.0.2.1/24) (VLAN 2)

CE1:
CUST1-Subnet: (198.51.100.1/24)

In this topology:¶

• BE1 (ESI-1): Shared by CUST1-VRF and CUST2-VRF with sub-interfaces VLAN 1 and 2¶

• BE2 (ESI-2): Used only by CUST1-VRF on native VLAN¶

To synchronize state for CUST1-VRF, the solution uses:¶

Case 1 - Native interface (BE2 to CE1):¶

• IP-VRF RT(s): Identifies CUST1-VRF¶

• ESI-2: Identifies BE2 interface¶

• Ethernet Tag-ID 0: Indicates native VLAN¶

Case 2 - Sub-interface (BE1.1 to SW1):¶

• IP-VRF RT(s): Identifies CUST1-VRF¶

• ESI-1: Identifies BE1 interface¶

• Ethernet Tag-ID 1: Identifies VLAN 1 sub-interface¶

3.2. Route Target Usage

Route synchronization between peering PEs uses EVPN route types as defined in [RFC7432] and [RFC9136].¶

Routes SHOULD be advertised with the ES-Import Route Target to identify the Layer-3 interface for which the information must be synchronized, and with the EVI-RT Extended Community [RFC9251] to identify the routing context (IP-VRF or GRT) in which synchronization occurs. This limits route distribution to PEs attached to the same ESI and ensures that the routes are applied to the correct IP-VRF at the receiving PE.¶

Alternatively, synchronization routes MAY be advertised using the IP-VRF route Targets. However, this approach may cause the routes to be distributed to all remote PEs in the IP-VRF, including those that do not require the synchronization information.¶

In Figure 3, CUST1 routes carry IP-VRF1 RT(s) and CUST2 routes carry IP-VRF2 RT(s). When using ES-Import RT optimization, routes also carry EVI-RT Extended Community with the corresponding IP-VRF RT.¶

Note: When VRF Route Targets are used, routes are distributed to all PEs importing that VRF RT, not just ESI-attached PEs. However, only PEs with EVPN SAFI enabled will process these routes, effectively limiting distribution to EVPN-capable PEs.¶

3.3. EVPN Instance Usage

Unlike [RFC7432] MAC-VRF deployments, EVI is not required for L3 multi-homing scenarios. The Route Distinguisher (RD) MAY be auto-generated locally, and Route Targets are taken from the IP-VRF configuration.¶

For Global Routing Table (GRT) services, an EVPN instance MAY be assigned to provide Route Targets as required by [RFC7432]. Alternatively, users MAY explicitly configure Route Targets for GRT synchronization.¶

The solution synchronizes the following state types:¶

• ARP/ND adjacencies (RT-2)¶

• IGMP/MLD join/leave (RT-7/RT-8)¶

• IGP learned routes (RT-5)¶

3.4. Mapping for L3 Interface to ESI

The ESI represents the L3 LAG interface between PE and CEs. This ESI is signaled using RT-4 with the ES-Import Route Target as described in Section 8.1.1 of [RFC7432] so that the service PE peers can discover each other's common ES.¶

In the example Figure 3, route-syncs from interface BE1 have IP-VRF RT(s) or ES-Import RT and EVI-RT EC with ESI 1 as an optimization.¶

3.5. Mapping for L3 Sub-Interface to Ethernet Tag-id

The Ethernet Tag-id represents the sub-interface subnet on the L3 LAG interface between PE and CEs. This apply to all route-sync types used for L3 multi-homing i.e., RT-2, RT-5, RT-7 and RT-8.¶

The Ethernet Tag ID encoded in synchronization routes is automatically derived from the encapsulation VLAN tags of the Layer-3 interface, following the encoding rules for single and double normalized VLAN identifiers defined in [RFC9744], as described below:¶

• Untagged Layer-3 LAG interfaces use an Ethernet Tag ID value of zero.¶

• Singly tagged Layer-3 LAG interfaces encode a single normalized VLAN identifier (VID) in the lower 12 bits of the Ethernet Tag ID field.¶

• Doubly tagged Layer-3 LAG interfaces encode the outer normalized VID in the upper 12 bits and the inner normalized VID in the lower 12 bits of the Ethernet Tag ID field.¶

For synchronization to operate correctly, PEs attached to the same multi-homed CE MUST use consistent VLAN identifiers for the same multihomed CE.¶

In the example Figure 3, route-syncs from sub-interface BE1.1 (VLAN1) is represented by Ethernet Tag Identifier with ID 1.¶

3.6. ARP/ND Synchronization

This section describes procedures for synchronizing ARP/ND adjacencies between PEs using EVPN RT-2 routes, as defined in Section 10 of [RFC7432], with modifications for Layer 3 interfaces.¶

3.7. IGMP/MLD Synchronization

This section describes procedures for synchronizing IGMP/MLD join and leave messages between PEs using EVPN RT-7 and RT-8 routes as defined in [RFC9251].¶

3.8. IGP Route Synchronization

This section describes procedures for synchronizing IGP learned customer routes between PEs using EVPN RT-5 routes as defined in Section 3 of [RFC9136].¶

When a CE forms an IGP adjacency on the LAG bundle, the adjacency may form to only one PE. That PE learns customer routes via IGP and must synchronize them to peer PEs so that all PEs can advertise the routes to the core and provide load-balancing.¶

Two approaches are defined:¶

1. ESI-based approach¶

2. IP Gateway-based approach¶