Benchmarking Methodology (bmwg) ------------------------------- Charter Last Modified: 2008-08-21 Current Status: Active Working Group Chair(s): Al Morton Operations and Management Area Director(s): Dan Romascanu Ronald Bonica Operations and Management Area Advisor: Ronald Bonica Mailing Lists: General Discussion:bmwg@ietf.org To Subscribe: bmwg-request@ietf.org In Body: subscribe your_email_address Archive: http://www.ietf.org/mail-archive/web/bmwg/index.html Description of Working Group: The major goal of the Benchmarking Methodology Working Group is to make a series of recommendations concerning the measurement of the performance characteristics of various internetworking technologies; further, these recommendations may focus on the systems or services that are built from these technologies. Each recommendation will describe the class of equipment, system, or service being addressed; discuss the performance characteristics that are pertinent to that class; clearly identify a set of metrics that aid in the description of those characteristics; specify the methodologies required to collect said metrics; and lastly, present the requirements for the common, unambiguous reporting of benchmarking results. To better distinguish the BMWG from other measurement initiatives in the IETF, the scope of the BMWG is limited to technology characterization using simulated stimuli in a laboratory environment. Said differently, the BMWG does not attempt to produce benchmarks for live, operational networks. Moreover, the benchmarks produced by this WG shall strive to be vendor independent or otherwise have universal applicability to a given technology class. Because the demands of a particular technology may vary from deployment to deployment, a specific non-goal of the Working Group is to define acceptance criteria or performance requirements. An ongoing task is to provide a forum for discussion regarding the advancement of measurements designed to provide insight on the operation inter-networking technologies. In addition to its current work plan, the BMWG is explicitly tasked to develop benchmarks and methodologies for the following technologies: * MPLS Forwarding: Develop specific methods to characterize the latency and forwarding performance of MPLS devices, extending the fundamental recommendations of RFC 1242 and RFC 2544 to this networking technology. * SIP Networking Devices: Develop new terminology and methods to characterize the key performance aspects of network devices using SIP, including the signaling plane scale and service rates while considering load conditions on both the signaling and media planes. This work will be harmonized with related SIP performance metric definitions prepared by the PMOL working group. Goals and Milestones: Done Expand the current Ethernet switch benchmarking methodology draft to define the metrics and methodologies particular to the general class of connectionless, LAN switches. Done Edit the LAN switch draft to reflect the input from BMWG. Issue a new version of document for comment. If appropriate, ascertain consensus on whether to recommend the draft for consideration as an RFC. Done Take controversial components of multicast draft to mailing list for discussion. Incorporate changes to draft and reissue appropriately. Done Submit workplan for initiating work on Benchmarking Methodology for LAN Switching Devices. Done Submit workplan for continuing work on the Terminology for Cell/Call Benchmarking draft. Done Submit initial draft of Benchmarking Methodology for LAN Switches. Done Submit Terminology for IP Multicast Benchmarking draft for AD Review. Done Submit Benchmarking Terminology for Firewall Performance for AD review Done Progress ATM benchmarking terminology draft to AD review. Done Submit Benchmarking Methodology for LAN Switching Devices draft for AD review. Done Submit first draft of Firewall Benchmarking Methodology. Done First Draft of Terminology for FIB related Router Performance Benchmarking. Done First Draft of Router Benchmarking Framework Done Progress Frame Relay benchmarking terminology draft to AD review. Done Methodology for ATM Benchmarking for AD review. Done Terminology for ATM ABR Benchmarking for AD review. Done Terminology for FIB related Router Performance Benchmarking to AD review. Done Firewall Benchmarking Methodology to AD Review Done First Draft of Methodology for FIB related Router Performance Benchmarking. Done First draft Net Traffic Control Benchmarking Methodology. Done Methodology for IP Multicast Benchmarking to AD Review. Done Resource Reservation Benchmarking Terminology to AD Review Done First I-D on IPsec Device Benchmarking Terminology Done EGP Convergence Benchmarking Terminology to AD Review Done Resource Reservation Benchmarking Methodology to AD Review Done Net Traffic Control Benchmarking Terminology to AD Review Done IGP/Data-Plane Terminology I-D to AD Review Done IGP/Data-Plane Methodology and Considerations I-Ds to AD Review Done Hash and Stuffing I-D to AD Review Done IPv6 Benchmarking Methodology to AD Review Sep 2008 IPsec Device Benchmarking Terminology to IESG Review Sep 2008 IPsec Device Benchmarking Methodology to IESG Review Dec 2008 Net Traffic Control Benchmarking Methodology to AD Review. Dec 2008 Router Accelerated Test Terminology to IESG Review Dec 2008 Router Accelerated Test Methodology to IESG Review Feb 2009 Terminology For Protection Benchmarking to AD Review Feb 2009 Methodology For Protection Benchmarking to AD Review Apr 2009 Methodology for MPLS Forwarding to AD Review Jun 2009 Terminology for SIP Device Benchmarking to IESG Review Jun 2009 Methodology for SIP Device Benchmarking to IESG Review Dec 2009 Router Accelerated Test Method for EBGP to IESG Review Dec 2009 Router Accelerated Test Method for Operational Security to IESG Review Jul 2010 Basic BGP Convergence Benchmarking Methodology to AD Review. Internet-Drafts: Posted Revised I-D Title ------ ------- -------------------------------------------- Jun 2003 Oct 2008 Benchmarking Methodology for Link-State IGP Data Plane Route Convergence Jun 2003 Oct 2008 Terminology for Benchmarking Link-State IGP Data Plane Route Convergence Jun 2003 Oct 2008 Considerations for Benchmarking Link-State IGP Data Plane Route Convergence Sep 2008 Sep 2008 MPLS Forwarding Benchmarking Methodology Request For Comments: RFC Stat Published Title ------- -- ----------- ------------------------------------ RFC1242 I Jul 1991 Benchmarking Terminology for Network Interconnection Devices RFC1944 I May 1996 Benchmarking Methodology for Network Interconnect Devices RFC2285 I Feb 1998 Benchmarking Terminology for LAN Switching Devices RFC2432 I Oct 1998 Terminology for IP Multicast Benchmarking RFC2544 I Mar 1999 Benchmarking Methodology for Network Interconnect Devices RFC2647 I Aug 1999 Benchmarking Terminology for Firewall Performance RFC2761 I Feb 2000 Terminology for ATM Benchmarking RFC2889 I Aug 2000 Benchmarking Methodology for LAN Switching Devices RFC3116 I Jun 2001 Methodology for ATM Benchmarking RFC3133 I Jun 2001 Terminology for Frame Relay Benchmarking RFC3134 I Jun 2001 Terminology for ATM ABR Benchmarking RFC3222 I Dec 2001 Terminology for Forwarding Information Base (FIB) based Router Performance RFC3511 I Apr 2003 Benchmarking Methodology for Firewall Performance RFC3918 I Oct 2004 Methodology for IP Multicast Benchmarking RFC4098 I Jun 2005 Terminology for Benchmarking BGP Device Convergence in the Control Plane RFC4061 I Jun 2005 Benchmarking Basic OSPF Single Router Control Plane Convergence RFC4062 I Jun 2005 OSPF Benchmarking Terminology and Concepts RFC4063 I Jun 2005 Considerations When Using Basic OSPF Convergence Benchmarks RFC4689 I Oct 2006 Terminology for Benchmarking Network-layer Traffic Control Mechanisms RFC4814 I Mar 2007 Hash and Stuffing: Overlooked Factors in Network Device Benchmarking RFC4883 I Jul 2007 Benchmarking Terminology for Resource Reservation Capable Routers RFC5180 I May 2008 IPv6 Benchmarking Methodology for Network Interconnect Devices