RE: Comments for Benchmarking Methodology for LAN Switching = Devices

From owner-bmwg@ironbridgenetworks.com Sat Aug 5 10:09:41 2000 Return-Path: Received: from ironbridgenetworks.com (helios.ironbridgenetworks.com [146.115.140.2]) by dokka.maxware.no (8.9.3/8.9.3) with ESMTP id KAA14415 for ; Sat, 5 Aug 2000 10:09:39 +0200 Received: (from majordom@localhost) by ironbridgenetworks.com (8.9.3/8.9.3) id EAA29764 for x-bmwg-include; Sat, 5 Aug 2000 04:06:50 -0400 (EDT) Received: from exchange1.netcomsystems.com (mushroom.netcomsystems.com [12.9.24.195]) by ironbridgenetworks.com (8.9.3/8.9.3) with ESMTP id EAA29758 for ; Sat, 5 Aug 2000 04:06:47 -0400 (EDT) Received: by exchange1.netcomsystems.com with Internet Mail Service (5.5.2650.21) id ; Sat, 5 Aug 2000 01:06:15 -0700 Message-ID: <9484475DFC05D2118F9C00805F6F263113790402@exchange1.netcomsystems.com> From: "Perser, Jerry" To: "BMWG (E-mail)" Subject: RE: Comments for Benchmarking Methodology for LAN Switching Devic es Date: Sat, 5 Aug 2000 01:06:11 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01BFFEB4.058C9B54" Sender: owner-bmwg@ironbridgenetworks.com Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BFFEB4.058C9B54 Content-Type: text/plain; charset="iso-8859-1" Gene, I wish the person going to the IETF meetings was keeping you updated better this last year (or two). The draft went to last call in March. The working group's rough consensus was to forward it to the AD. It was approved by the IESG last June. It is currently waiting for the RFC editor to finish it. Since your comments did not have any technical merit, I do not want to pull back the draft to address your issues. I beleive the working group's time would be better spent on other items than how many times the word "the" appears in a paragraph, if a sentence has a noun AND a verb, or if " I before E except after C". Going forward, here is what I propose: A. When the RFC Editor gives me 48 hours for any last minute changes, I will fix the reference to RFC 2544. You were right that it was not published in 2000. This is not a technical change. B. Set up a face to face meeting to address your issues. We work within walking distance of each other. I can then explain why your issues that appear technical were wrong. C. Any comments that do not need the working group to discuss, to be addressed directly to me. I do respond to direct input (other WG members can attest to this). Bring this type of issues to the reflector only if the author ignores you (like Debbie did with the comments I personally gave here in Washington DC). Jerry Perser ------_=_NextPart_001_01BFFEB4.058C9B54 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: Comments for Benchmarking Methodology for LAN Switching = Devices

Gene,

I wish the person going to the IETF meetings was = keeping you updated better this last year (or two). The draft = went to last call in March. The working group's rough consensus = was to forward it to the AD. It was approved by the IESG last = June. It is currently waiting for the RFC editor to finish = it.

Since your comments did not have any technical merit, = I do not want to pull back the draft to address your issues. I = beleive the working group's time would be better spent on other items = than how many times the word "the" appears in a paragraph, if = a sentence has a noun AND a verb, or if " I before E except after = C".

Going forward, here is what I propose:

A. When the RFC Editor gives me 48 hours for any last = minute changes, I will fix the reference to RFC 2544. You were = right that it was not published in 2000. This is not a technical = change.

B. Set up a face to face meeting to address your = issues. We work within walking distance of each other. I = can then explain why your issues that appear technical were = wrong.

C. Any comments that do not need the working group to = discuss, to be addressed directly to me. I do respond to direct = input (other WG members can attest to this). Bring this type of = issues to the reflector only if the author ignores you (like Debbie did = with the comments I personally gave here in Washington DC).

Jerry Perser

------_=_NextPart_001_01BFFEB4.058C9B54-- From owner-bmwg@ironbridgenetworks.com Wed Aug 9 02:03:57 2000 Return-Path: Received: from ironbridgenetworks.com (helios.ironbridgenetworks.com [146.115.140.2]) by dokka.maxware.no (8.9.3/8.9.3) with ESMTP id CAA18433 for ; Wed, 9 Aug 2000 02:03:55 +0200 Received: (from majordom@localhost) by ironbridgenetworks.com (8.9.3/8.9.3) id UAA19121 for x-bmwg-include; Tue, 8 Aug 2000 20:00:58 -0400 (EDT) Received: from ixiacom.com ([63.82.51.3]) by ironbridgenetworks.com (8.9.3/8.9.3) with ESMTP id UAA19108 for ; Tue, 8 Aug 2000 20:00:56 -0400 (EDT) Received: from [192.168.3.29] (HELO thing1) by ixiacom.com (CommuniGate Pro SMTP 3.1) with SMTP id 2790918 for bmwg@ironbridgenetworks.com; Tue, 08 Aug 2000 17:01:49 -0700 From: "Debby Stopp" To: "bmwg" Subject: FW: Comments for Benchmarking Methodology for LAN Switching Devices Date: Tue, 8 Aug 2000 17:02:20 -0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-bmwg@ironbridgenetworks.com Precedence: bulk Hi Jerry, The Comments emails were a collaborative effort. The feedback was collected from multiple sources and forwarded by myself. I don't get the "no technical merit" part - I did take another look and I don't think that it's out of line to point out the divide by zero error in Appendix A, for example . Besides, the WG is a bunch of scientists so certainly it's in the spirit of things to hear all comments, valid or not. If criticism doesn't hold water, then this will stand out (and the same goes for proposals). You make a valid point that the comments came after last call, however comments should be always be welcome. Best Regards, - Gene From owner-bmwg@ironbridgenetworks.com Fri Aug 25 17:48:23 2000 Return-Path: Received: from ironbridgenetworks.com (helios.ironbridgenetworks.com [146.115.140.2]) by dokka.maxware.no (8.9.3/8.9.3) with ESMTP id RAA04608 for ; Fri, 25 Aug 2000 17:48:19 +0200 Received: (from majordom@localhost) by ironbridgenetworks.com (8.9.3/8.9.3) id LAA04855 for x-bmwg-include; Fri, 25 Aug 2000 11:44:06 -0400 (EDT) Received: from exchange1.netcomsystems.com (mushroom.netcomsystems.com [12.9.24.195]) by ironbridgenetworks.com (8.9.3/8.9.3) with ESMTP id LAA04818 for ; Fri, 25 Aug 2000 11:44:03 -0400 (EDT) Received: by exchange1.netcomsystems.com with Internet Mail Service (5.5.2650.21) id ; Fri, 25 Aug 2000 08:43:32 -0700 Message-ID: <9384475DFC05D2118F9C00805F6F2631018A54E8@exchange1.netcomsystems.com> From: "Perser, Jerry" To: "BMWG (E-mail)" Subject: Workplan Proposal: Benchmarking Differentiated Services Mechanism - Version 2 Date: Fri, 25 Aug 2000 08:43:26 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C00EAB.36CF4F2A" Sender: owner-bmwg@ironbridgenetworks.com Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C00EAB.36CF4F2A Content-Type: text/plain; charset="iso-8859-1" Workplan Proposal: Benchmarking Differentiated Services Mechanisms J. Perser, Spirent Communications D. Newman, Network Test S. Poretsky, Avici July 2000 This memo proposes the development of practices for benchmarking the performance of devices that implement differentiated services mechanisms. The objective of such mechanisms is to prioritize certain traffic classes during periods of device or network congestion. Accordingly, the objective of this effort is to develop terminology and methodology for evaluating the effectiveness of those mechanisms. We propose that this effort be conducted within the Benchmarking Working Group (bmwg) of the Internet Engineering Task Force (IETF). 1. Effort's Motivation Driven by exponential traffic growth, demand for bandwidth-hungry applications, and a desire for guaranteed service levels, vendors of Internet routers and switches had added mechanisms intended to ensure different service levels for different classes of IP traffic. There is currently no standard practice for evaluating whether these Differentiated Services (DS) mechanisms actually work as advertised. As a result, there are widely varying definitions of terms and methodologies for evaluating DS performance. The lack of an agreed-upon terminology and methodology makes it difficult to perform meaningful comparisons of different vendors' implementations of the same mechanism. 2. Effort's Goals The general goal of this effort is to develop terminology and methodology documents for benchmarking the per-hop behavior characteristics of devices implementing DS mechanisms. The specific focus of this effort is to identify terms and practices for benchmarking devices that examine either the precedence field as defined in RFC 791 or the differentiated services code point as defined in RFC 2474. The authors' intent with these documents is to enable comparison of routing/switching devices that use DS enabled hardware or software. 3. Plan and Milestones This effort will follow the standard BMWG procedure of producing separate terminology and methodology documents. 3.1 Terminology Document The terminology document will serve as the foundation for the methodology document by defining DS test metrics and other concepts related to IP precedence and DSCP settings. The main metrics to be addressed in the terminology document include: --forwarding rate --packet loss --latency --jitter The terminology document also will define concepts and parameters for a congesting a network. Differentiated services may not become active until the traffic exceeds the DUT/SUT throughput. Describing how a test congests the network is as important as how the device under test prioritizes traffic during periods of congestion. 3.1.1 Terminology Milestones We propose the following timetable for completion of the terminology document: October 2000 First draft presented to BMWG April 2001 WG Last Call on DS Benchmarking Terminology I-D. 3.2 Methodology Documents After the BMWG reaches rough consensus on the terminology document, the authors propose to write companion methodology document. The authors propose a single methodology document for measuring performance of devices using IP precedence and DSCP field mechanisms. If a rough consensus can not be made on a single document, then separate methodologies will be written for each of the two mechanisms. 3.2.1 Methodology Milestones We propose the following timetable for completion of the DS test methodology documents. March 2001 First draft of Benchmarking Methodology July 2001 Consensus on a single Methodology document November 2001 WG Last Call on Benchmarking Methodology ___________________________________________________________ Jerry Perser Phone: 818.676.2320 SmartLab Manager Lab: 818.676.2337 SmartBits Systems Fax: 818.880.9165 26750 Agoura Road Corp: 818.676.2300 Calabasas, CA, 91302 Pager: 8185963000.0089988@pagenet.net ------_=_NextPart_001_01C00EAB.36CF4F2A Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Workplan Proposal: Benchmarking Differentiated Services = Mechanism - Version 2

Workplan Proposal: Benchmarking Differentiated = Services Mechanisms
J. Perser, Spirent Communications
D. Newman, Network Test
S. Poretsky, Avici

July 2000

This memo proposes the development of practices for = benchmarking the performance of devices that implement differentiated = services mechanisms. The objective of such mechanisms is to prioritize = certain traffic classes during periods of device or network congestion. = Accordingly, the objective of this effort is to develop terminology and = methodology for evaluating the effectiveness of those = mechanisms.

We propose that this effort be conducted within the = Benchmarking Working Group (bmwg) of the Internet Engineering Task = Force (IETF).

1. Effort's Motivation

Driven by exponential traffic growth, demand for = bandwidth-hungry applications, and a desire for guaranteed service = levels, vendors of Internet routers and switches had added mechanisms = intended to ensure different service levels for different classes of IP = traffic. There is currently no standard practice for evaluating = whether these Differentiated Services (DS) mechanisms actually work as = advertised. As a result, there are widely varying definitions of = terms and methodologies for evaluating DS performance.

The lack of an agreed-upon terminology and = methodology makes it difficult to perform meaningful comparisons of = different vendors' implementations of the same mechanism.

2. Effort's Goals

The general goal of this effort is to develop = terminology and methodology documents for benchmarking the per-hop = behavior characteristics of devices implementing DS = mechanisms.

The specific focus of this effort is to identify = terms and practices for benchmarking devices that examine either the = precedence field as defined in RFC 791 or the differentiated services = code point as defined in RFC 2474.

The authors' intent with these documents is to enable = comparison of routing/switching devices that use DS enabled hardware or = software.

3. Plan and Milestones

This effort will follow the standard BMWG procedure = of producing separate terminology and methodology documents.

3.1 Terminology Document

The terminology document will serve as the foundation = for the methodology document by defining DS test metrics and other = concepts related to IP precedence and DSCP settings. The main = metrics to be addressed in the terminology document include:

--forwarding rate
--packet loss
--latency
--jitter

The terminology document also will define concepts = and parameters for a congesting a network. Differentiated = services may not become active until the traffic exceeds the DUT/SUT = throughput. Describing how a test congests the network is as = important as how the device under test prioritizes traffic during = periods of congestion.

3.1.1 Terminology Milestones

We propose the following timetable for completion of = the terminology document:

October 2000 First = draft presented to BMWG
April = 2001 WG Last Call on DS Benchmarking Terminology = I-D.

3.2 Methodology Documents

After the BMWG reaches rough consensus on the = terminology document, the authors propose to write companion = methodology document. The authors propose a single methodology = document for measuring performance of devices using IP precedence and = DSCP field mechanisms. If a rough consensus can not be made on a = single document, then separate methodologies will be written for each = of the two mechanisms.

3.2.1 Methodology Milestones

We propose the following timetable for completion of = the DS test methodology documents.

March    2001    = First draft of Benchmarking Methodology
July     = 2001    Consensus on a single Methodology = document
November 2001    WG Last Call = on Benchmarking Methodology

___________________________________________________________ =
Jerry = Perser=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Phone: 818.676.2320 =
SmartLab = Manager=A0=A0=A0=A0=A0 Lab: = 818.676.2337
SmartBits Systems = =A0=A0=A0=A0=A0=A0=A0=A0Fax:=A0=A0=A0818.880.9165=A0=A0=A0=A0
26750 Agoura = Road=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Corp: 818.676.2300
Calabasas, CA, = 91302 = Pager:=A08185963000.0089988@pagenet.net

=A0

------_=_NextPart_001_01C00EAB.36CF4F2A-- From owner-bmwg@ironbridgenetworks.com Fri Aug 25 17:51:19 2000 Return-Path: Received: from ironbridgenetworks.com (helios.ironbridgenetworks.com [146.115.140.2]) by dokka.maxware.no (8.9.3/8.9.3) with ESMTP id RAA04671 for ; Fri, 25 Aug 2000 17:51:17 +0200 Received: (from majordom@localhost) by ironbridgenetworks.com (8.9.3/8.9.3) id LAA09952 for x-bmwg-include; Fri, 25 Aug 2000 11:50:07 -0400 (EDT) Received: from exchange1.netcomsystems.com (mushroom.netcomsystems.com [12.9.24.195]) by ironbridgenetworks.com (8.9.3/8.9.3) with ESMTP id LAA09849 for ; Fri, 25 Aug 2000 11:50:03 -0400 (EDT) Received: by exchange1.netcomsystems.com with Internet Mail Service (5.5.2650.21) id ; Fri, 25 Aug 2000 08:49:32 -0700 Message-ID: <9384475DFC05D2118F9C00805F6F2631018A54EA@exchange1.netcomsystems.com> From: "Perser, Jerry" To: "'BMWG (E-mail)'" Subject: Methodology for IP Mulicast Benchmarking - Scaled Group Forwardin g Matrix Date: Fri, 25 Aug 2000 08:49:26 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C00EAC.0DBD8C86" Sender: owner-bmwg@ironbridgenetworks.com Precedence: bulk This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C00EAC.0DBD8C86 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I am a little confused on the procedure for ports joining after = multicast traffic is sent. The last two paragraphs in the procedure talk about = how the ports continue to join incremental. Can we change it to state how = to start, then how to continue? And can we use the term "destination = port" as used in section 3 (only if you really meant destination not receive)? In the results, can we add that forwarding rate MUST be measured? The requirement for frame loss, rates, and frame counts should be set = to MAY. These measurements are not used in calculating the Scaled Group Forwarding Matrix. They are used for debugging purposes. If you think = the Scaled Group Forwarding Matrix is in error, you MAY want to look at the frame loss, frame rates, and frame counts. Also, can we use the terminology from RFC 2285 instead of transmit rate = and receive rate? ___________________________________________________________=20 Jerry Perser=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Phone: = 818.676.2320=20 SmartLab Manager=A0=A0=A0=A0=A0 Lab: 818.676.2337=20 SmartBits Systems = =A0=A0=A0=A0=A0=A0=A0=A0Fax:=A0=A0=A0818.880.9165=A0=A0=A0=A0=20 26750 Agoura Road=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0Corp: 818.676.2300=20 Calabasas, CA, 91302 Pager:=A08185963000.0089988@pagenet.net =A0=20 ------_=_NextPart_001_01C00EAC.0DBD8C86 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Methodology for IP Mulicast Benchmarking - Scaled Group = Forwarding Matrix

I am a little confused on the procedure for ports = joining after multicast traffic is sent. The last two paragraphs = in the procedure talk about how the ports continue to join = incremental. Can we change it to state how to start, then how to = continue? And can we use the term "destination port" as = used in section 3 (only if you really meant destination not = receive)?

In the results, can we add that forwarding rate MUST = be measured?

The requirement for frame loss, rates, and frame = counts should be set to MAY. These measurements are not used in = calculating the Scaled Group Forwarding Matrix. They are used for = debugging purposes. If you think the Scaled Group Forwarding = Matrix is in error, you MAY want to look at the frame loss, frame = rates, and frame counts.

Also, can we use the terminology from RFC 2285 = instead of transmit rate and receive rate?

=A0

------_=_NextPart_001_01C00EAC.0DBD8C86--