<?xml version="1.0" encoding="utf-8"?>
<?xml-model href="rfc7991bis.rnc"?>

<!DOCTYPE rfc [
  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;">
]>
<rfc
  xmlns:xi="http://www.w3.org/2001/XInclude"
  category="std"
  docName="draft-ietf-idr-next-next-hop-nodes-02"
  ipr="trust200902"
  obsoletes=""
  updates=""
  submissionType="IETF"
  xml:lang="en"
  version="3"
  consensus="true">

  <front>
    <title abbrev="NNHN">BGP Next-next Hop Nodes</title>

    <seriesInfo name="Internet-Draft" value="draft-ietf-idr-next-next-hop-nodes-02"/>
   
    <author fullname="Kevin Wang" initials="K" surname="Wang">
      <organization>HPE</organization>
      <address>
        <email>kevin.wang@hpe.com</email>  
      </address>
    </author>
   
    <author fullname="Jeff Haas" initials="J" surname="Haas">
      <organization>HPE</organization>
      <address>
        <email>jeffrey.haas@hpe.com</email>  
        <uri></uri>
      </address>
    </author>
   
    <author fullname="Changwang Lin" initials="C" surname="Lin">
      <organization>New H3C Technologies</organization>
      <address>
        <email>linchangwang.04414@h3c.com</email>  
        <uri></uri>
      </address>
    </author>
      
    <author fullname="Jeff Tantsura" initials="J" surname="Tantsura">
      <organization>Nvidia</organization>
      <address>
        <email>jefftant.ietf@gmail.com</email>  
        <uri></uri>
      </address>
    </author>

    <date year="2026"/>
    <area>Routing</area>
    <workgroup>IDR</workgroup>

    <keyword>BGP</keyword>

    <abstract>
      <t>
        BGP speakers learn their next-hop addresses for NLRI in RFC 4271
        in the NEXT_HOP field and in RFC 4760
        in the "Network Address of Next Hop" field.
	Under certain circumstances, it might
	be desirable for a BGP speaker to know both the next-hops and the next-next-hops of 
        NLRI to make optimal forwarding decisions.  One such example is
        global load balancing (GLB) in a Clos network.
      </t>

      <t>
	Draft-ietf-idr-nhc defines the "Next Hop Dependent
	Characteristics Attribute" (NHC) which allows a BGP speaker to signal the forwarding
	characteristics associated with a given next-hop.
      </t>
      
      <t>
	This document defines a new NHC characteristic, the
	Next-next Hop Nodes (NNHN) characteristic, which can be used to advertise
	the next-next-hop nodes	associated with a given next-hop.
      </t>
    </abstract>
 
  </front>

  <middle>
    
    <section anchor="Introduction">
      <name>Introduction</name>
      <t>
        BGP speakers learn their next-hop addresses for NLRI in <xref target="RFC4271"/> 
        in the NEXT_HOP field and in <xref target="RFC4760"/>
        in the "Network Address of Next Hop" field.
	Under certain circumstances, it might
	be desirable for a BGP speaker to know both the next-hops and the next-next-hops of 
        NLRI to make optimal forwarding decisions.  One such example is the
        global load balancing (GLB) in a Clos network <xref target="I-D.cheng-rtgwg-adaptive-routing-framework"/>.
      </t>

      <t>
        When a route's ECMP has multiple next-hops, packets forwarded using
        that ECMP are hashed to the member next-hops for load balancing purposes. If
        one of the member next-hop links is congested due
	to uneven hashing, dynamic load balancing (DLB) allows the node to adjust the hashing
	so that the congestion on that link can be mitigated. When all next-hop
	link(s) are congested, DLB on the local node will not help to
	mitigate the congestion. Such nodes will require help from the previous hop to shift
	the traffic towards alternative nodes to mitigate such congestion.
	This process is called global load balancing.
      </t>

      <t>
	In a Clos network, a congested link will affect the
	load balancing decisions of the previous layer nodes equally. Because of this, the
	previous-previous layer nodes do not need to change their load balancing decisions
	towards the previous layer nodes to mitigate this link congestion. This means we
	only need to know the link congestion status of the next-next-hops of given BGP route
	in order to make GLB decisions. The combined link quality (congestion/failure) of each next-hop and its
	corresponding next-next-hops can be used as the feedback for DLB.
      </t>
      <t>
        The purpose of this document is to provide a method for BGP to learn
        the next-next-hops, or more specifically, the next-next-hop
        nodes. When a next-hop node has parallel links towards a
        next-next-hop node, DLB helps to balance the load between the parallel links
	by locally adjusting the volume of traffic hashed over a
        given ECMP member link. Thus, only the overall link congestion between
        the next-hop node and the next-next-hop node is important for GLB.
      </t>
      <t>
        Note that the mechanism for detecting link congestion and communicating
        them to the previous hop nodes is out of the scope of this document.
	<xref target="I-D.zzhang-fann-router-info"/> defined one approach
	to notify link quality to directly connected nodes.
      </t>

      <t>
        This document defines a new NHC characteristic, the Next-next Hop Nodes
        (NNHN) characteristic, for the BGP Next Hop Dependent Characteristics
        Attribute (NHC) defined in 
        <xref target="I-D.ietf-idr-nhc"/>.
	A downstream BGP speaker can use the NNHN to advertise the
	next-next-hop nodes corresponding to the next-hop of an NLRI.
	This allows the upstream BGP speaker to learn the next-next-hop nodes
	corresponding to each of its next-hop nodes.
      </t>
      <section>
        <name>Requirements Language</name>
        <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL",
          "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
          RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
          interpreted as described in BCP 14 <xref target="RFC2119"/>
          <xref target="RFC8174"/> when, and only when, they appear in
          all capitals, as shown here.</t>
      </section>

    </section>
    
    <section>
      <name>BGP Next-next Hop Nodes (NNHN) Characteristic</name>

      <t>
        <xref target="I-D.ietf-idr-nhc"/> defines NHC as a container
        for characteristic TLVs. Next-next Hop Nodes is one such characteristic. It
        specifies the next-next-hop nodes corresponding to the next-hop field
        in the NHC.
      </t>
      
      <section>
	<name>Encoding NNHN</name>
	
	<t>
          The NNHN TLV has the NHC characteristic code 2, as assigned in
	  <xref target="I-D.ietf-idr-nhc" sectionFormat="of" section="4"/>.
	  The NHC characteristic length is variable. The NNHN characteristic format is shown in
	  <xref target="nnh_attr"/>:
	</t>
      
	<figure anchor="nnh_attr">
          <name>NNHN Characteristic TLV Format</name>
          <artwork type="ascii-art">
            <![CDATA[
   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |    Characteristic Code = 2    |Characteristic Length(variable)|
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                    Next-hop BGP ID                            |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  ~               Next-next-hop BGP IDs (variable)                ~
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
            ]]>
          </artwork>
	</figure>

	<dl newline="true">
          <dt>
	    Next-hop BGP ID: 
	  </dt>
	  <dd>
	    32-bit BGP Identifier of the next-hop node attaching this NHC characteristic.
	  </dd>
          <dt>
	    Next-next-hop BGP IDs: 
	  </dt>
	  <dd>
	    One or more 32-bit BGP Identifiers, each representing a next-hop node used by
	    the node referenced in the "Next-hop BGP ID" for ECMP forwarding.
	  </dd>
	</dl>
      </section>

      <section>
	<name>Sending NNHN</name>

        <t>
          All procedures from 
          <xref target="I-D.ietf-idr-nhc" sectionFormat="of" section="2.2"/>
          apply.
        </t>
	
	<t>
          When a BGP speaker S has a BGP route R it wishes to advertise
          with next-hop self to its peer, it MAY choose to originate an NNHN
          characteristic. The "Next-hop BGP ID" field MUST be set to the BGP
          Identifier this BGP speaker uses with the peer.
	</t>

	<t>
	  If route R has only a single path, the BGP Identifier of the BGP peer that advertised the path
	  MUST be encoded in the "Next-next-hop BGP IDs" field.
	</t>
	
	<t>
          If route R has multiple paths, for all the ECMP paths which are used for forwarding, the
          BGP Identifiers of those BGP peers that advertised the ECMP paths MUST be encoded in the
          "Next-next-hop BGP IDs" field.
	</t>

	<t>
          When there are more than one "Next-next-hop BGP IDs" in the
          characteristic, they MUST be encoded in the numerically ascending order, with no duplicates,
          treating the BGP Identifier as a network byte order encoded 32-bit
          unsigned integer.
        </t>

        <t>
          An NNHN MUST have at least one "Next-next-hop BGP ID".
        </t>

	<t>
          When a BGP speaker S has a BGP route R it wishes to advertise with
          next-hop self to its peer, it MUST NOT forward the NNHN characteristic
          received from downstream peers. It either originates its own NNHN
          characteristic as described above or does not send one.
	</t>
	
	<t>
          When a BGP speaker S has a BGP route R it wishes to advertise
          with the next-hop that has not been set to self, it MUST NOT
          originate an NNHN characteristic.
          However, if a NNHN characteristic has been
          received for route R and passed the NHC validation as defined in
          <xref target="I-D.ietf-idr-nhc"/>, the NNHN
          characteristic SHOULD be forwarded.
	</t>

	<t>
	  A BGP speaker MUST NOT include more than one instance of NNHN in an NHC.
        </t>

	<section>
	  <name>Aggregation</name>
	  <t>
	    When the NNHN is originated locally due to next-hop self, there is no
	    case in which it needs to be aggregated. When the NNHN is not originated
	    locally because the next-hop is not set to self, we MUST NOT forward the NNHN
	    unless all constituent routes of the aggregate route have the same NNHN.
	  </t>
	</section>
      </section>
      
      <section>
	<name>Receiving NNHN</name>

        <t>
          All procedures from 
          <xref target="I-D.ietf-idr-nhc" sectionFormat="of" section="2.3"/>
          apply.
        </t>

	<t>
	  The receiver of the NNHN characteristic MUST be able to handle any order of the
          "Next-next-hop BGP IDs".
	</t>

	<t>
	  The receiver of the NNHN characteristic uses the "Next-hop BGP ID" and
	  "Next-next-hop BGP IDs" fields to determine the set of next-next-hops used
	  by the next-hop of this route. Usage of this information is beyond the scope of this
	  document. Please refer to <xref target="Introduction"/> for some of the usage examples.
	</t>
      </section>
      
      <section>
	<name>NNHN Error Handling</name>
	
        <t>
          The NNHN characteristic length MUST be at least 8 and MUST be a multiple
          of 4, otherwise it is malformed.  Malformed NNHN characteristics MUST be
          discarded.
        </t>

	<t>
	  If more than one instance of NNHN is included in an NHC, instances
	  beyond the first MUST be discarded.
        </t>

        <t>
          When an NNHN characteristic is received from a single-hop EBGP peer and the "Next-hop BGP Identifier"
          does not match the peer's BGP Identifier, it MUST be discarded.
        </t>

	<t>
          When duplicate "Next-next-hop BGP IDs" exist in an NNHN, only the first
	  one is accepted and the remaining duplicates MUST be discarded.
	</t>
     </section>
    </section>

    <section>
      <name>Operational Considerations</name>

      <t>
	NNHN characteristic is intended to be used under data center Clos network described in <xref target="RFC7938"/>.
	Since BGP Identifiers are used to identify the next-next-hop nodes, an operation
	MUST ensure that the BGP Identifiers are unique across the Clos network where NNHN characteristic
	is sent.
      </t>

      <t>
	In the event of a topology change, such as a link failure, propagating the NNHN characteristic
	may increase the number of BGP route updates.
	For example, in a five-stage Clos network, if a link between a spine and a leaf fails, the spine
	must withdraw the leaf route from the super spines. However, if a super spine advertises the
	NNHN characteristic for the leaf route, it must also send route updates to the other spines to
	reflect the change in NNHN, since the leaf route on the super spine has lost one of its ECMP
	next-hop legs.
	These additional route updates sent by the super spine to the other spines represent the
	overhead introduced by NNHN propagation.
	A typical super spine in a Clos network may connect to 32 spines. Considering that the number of
	routes received from any individual leaf is usually limited and that link failures within the network
	are relatively infrequent, the additional route updates resulting from NNHN propagation should
	be manageable.
      </t>
    </section>
    
    <section anchor="IANA">
      <name>IANA Considerations</name>
      <t>
	Since NHC Characteristic Code 2, has already been assigned in
	<xref target="I-D.ietf-idr-nhc" sectionFormat="of" section="5"/>,
	for the NNHN characteristic, there is no further IANA actions for this document.
      </t>
    </section>
    
    <section anchor="Implementation">
      <name>Implementation Status</name>
      <t>
        [The RFC-Editor should remove this section before publishing this
	document as an RFC]
      </t>

      <t>
	Global load balancing using NNHN characteristic has been implemented by
	HPE Networking Juniper and New H3C Technologies. Interoperation tests
	between both implementations were successfully conducted in 2024.
      </t>
    </section>
    
    <section anchor="Security">
      <name>Security Considerations</name>
      <t>
        As NNHN is not cryptographically protected, modification or insertion of a syntactically valid but
	bogus NNHN characteristic by an attacker would perturbate the applications using the knowledge
	of these next-next-hops.
      </t>
    </section>
  </middle>

  <back>
    <references>
      <name>References</name>
      <references>
        <name>Normative References</name>
        
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4271.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.draft-ietf-idr-nhc-07.xml"/>

      </references>
 
      <references>
        <name>Informative References</name>
       
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.7938.xml"/>
        <xi:include href="https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4760.xml"/>
	<xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.draft-cheng-rtgwg-adaptive-routing-framework-06.xml"/>
	<xi:include href="https://bib.ietf.org/public/rfc/bibxml3/reference.I-D.draft-zzhang-fann-router-info-00.xml"/>

      </references>
    </references>

    <section anchor="Acknowledgements" numbered="false">
      <name>Acknowledgements</name>
      <t>
        The authors would like to thank John Scudder, Jie Dong, Robert Raszuk, Jordan Head, Andrew Stone, Gyan Mishra for
	their reviews, comments, and suggestions. Special thanks to Bruno Decraene for his extremely
	thoughtful review and even writeup suggestions for some of the paragraphs.
      </t>
    </section>
    
 </back>
</rfc>
