Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa06221; 4 Dec 92 11:51 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa06217; 4 Dec 92 11:51 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa06144; 4 Dec 92 11:53 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.2) id AA25086; Fri, 4 Dec 92 10:53:12 CST Received: by hemlock.cray.com id AA09998; 4.1/CRI-5.5; Fri, 4 Dec 92 10:53:09 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA09988; 4.1/CRI-5.5; Fri, 4 Dec 92 10:53:05 CST Received: from relay2.UU.NET by cray.com (4.1/CRI-MX 2.2) id AA24977; Fri, 4 Dec 92 10:53:04 CST Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA19073; Fri, 4 Dec 92 11:53:06 -0500 Received: from s5000.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 115031.3559; Fri, 4 Dec 1992 11:50:31 EST Received: by unirsvl.RSVL.UNISYS.COM (smail2.5) id AA17030; 4 Dec 92 10:07:00 CST (Fri) Subject: Re: 50000 connections and gigabit speeds To: Rob Warnock Date: Fri, 4 Dec 92 10:06:58 CST Cc: tcplw@cray.com In-Reply-To: ; from "Rob Warnock" at Nov 29, 92 2:50 am X-Mailer: ELM [version 2.2 PL0] Message-Id: <9212041006.AA17026@unirsvl.RSVL.UNISYS.COM> Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: kurt@unirsvl.rsvl.unisys.com > > kurt@unirsvl.RSVL.UNISYS.COM says: > +--------------- > | > > It is highly unlikely that one would want to run > | > > 50,000 tcp connections at gigabit speeds (you would need a > | > > lot of buffer space). > | > | Why not? The gigabit speed would just decrease the network latency time, > | wouldn't it? Then you could get the same transaction rate on the host with > | fewer networks. The buffer space needed would not go up. I am interested in > | a transaction environment only, not doing 50,000 file transfers at the same > | time. > +--------------- > > Then you are going to be mightily disappointed in gigabit networks. Except > for some few cases of mongo-gram transactions in low-diameter LANs, you will > not find gigabit networks decreasing latency much. Consider: An 800 mile T3 > link (through two routers) is about 25 ms [as reported by skibo@cs.uiuc.edu, > now skibo@sgi.com]. That latency will not lessen (and it probably will even > even go up a bit) if you run over an OC-12 ATM link (622 Mb/s). Unless each > separate "transaction" is *huge* [which creates error & flow control problems], > an increase in link speed beyond double-digit megabits/sec will not buy you > much of anything. > > Even in a LAN, increases in link speed give disappointingly small decreases > in latency. Just ask anybody about NFS over Ethernet versus over FDDI... > > Gigabits are for blasting files, not transactions [unless you have 50,000 > outstanding transactions or something].... > > > -Rob > By this logic I should be using 2400 baud lines instead of Ethernet or FDDI? :-) No, seriously, am I missing something? Can you only get gigabit speeds out of WANs? Why can't we build a gigabit LAN if we can build a gigabit WAN? On another note, I think that we can drop the discussion of 50,000 connections in a transaction environment here. TCP large windows aren't going to be helpful in that situation anyway. We are getting sidetracked on an issue that doesn't really belong here. Kurt > ----- > Rob Warnock, MS-9U/510 rpw3@sgi.com > Silicon Graphics, Inc. (415)390-1673 > 2011 N. Shoreline Blvd. > Mountain View, CA 94043 > > > > >   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa17458; 7 Dec 92 13:02 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa17454; 7 Dec 92 13:02 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa13314; 7 Dec 92 13:03 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.2) id AA12221; Mon, 7 Dec 92 12:03:58 CST Received: by hemlock.cray.com id AA27586; 4.1/CRI-5.6; Mon, 7 Dec 92 12:03:53 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA27580; 4.1/CRI-5.6; Mon, 7 Dec 92 12:03:48 CST Received: from somnet.Sandia.GOV by cray.com (4.1/CRI-MX 2.2) id AA12204; Mon, 7 Dec 92 12:03:46 CST Received: by somnet.Sandia.GOV (5.59/SNLA-3.2.1) id AA22384; Mon, 7 Dec 92 11:01:14 MST Date: Mon, 7 Dec 92 11:01:14 MST Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: "C. Douglas Brown" Message-Id: <9212071801.AA22384@somnet.Sandia.GOV> To: tcplw@cray.com Subject: Latency and Gigabit speeds > By this logic I should be using 2400 baud lines instead of Ethernet or FDDI? :-) > No, seriously, am I missing something? Can you only get gigabit speeds out of > WANs? Why can't we build a gigabit LAN if we can build a gigabit WAN? > > On another note, I think that we can drop the discussion of 50,000 connections > in a transaction environment here. TCP large windows aren't going to be > helpful in that situation anyway. We are getting sidetracked on an issue that > doesn't really belong here. > > Kurt I don't think anyone is trying to say you can't build gigabit LANs, but Rob has a valid point that simply increasing the speed of a network (LAN or WAN) doesn't necessarily decrease its latency. In fact, most of the delays in a network remain pretty constant when you increase the bit rate, except for packet transmission time. This means that the TCP large windows will even be needed for gigabit LANs, as well as WANs, because a sender might fill the standard 64K window before an acknowledgement could be received . Doug Brown Sandia National Labs   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa10045; 8 Dec 92 16:09 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa10041; 8 Dec 92 16:09 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa26453; 8 Dec 92 16:10 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.2) id AA15152; Tue, 8 Dec 92 15:10:41 CST Received: by hemlock.cray.com id AA25214; 4.1/CRI-5.6; Tue, 8 Dec 92 15:10:38 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA25196; 4.1/CRI-5.6; Tue, 8 Dec 92 15:10:32 CST Received: from handel.cray.com.cray.com by cray.com (4.1/CRI-MX 2.2) id AA15101; Tue, 8 Dec 92 15:10:18 CST Received: by handel.cray.com.cray.com id AA20597; 5.65/CRI-5.5; Tue, 8 Dec 1992 15:07:40 -0600 Date: Tue, 8 Dec 1992 15:07:40 -0600 Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: "David A. Borman" Message-Id: <9212082107.AA20597@handel.cray.com.cray.com> To: tcplw@cray.com Subject: 1323 implementations Hi. It has now been 6 months since RFC 1323 (TCP Extensions for High Performance) was published, so it how now met the minimum time requirement before being allowed to be advanced from a Proposed Standard to a Draft Standard. Before it can be elevated to Draft, there have to be at least two independent and interoperable implementations. So, I am taking an implementation survey. If you check the box to keep the information confidential, then the only information that I will make public is that there is an implementation, and which parts of the RFC it supports. Names/Organization/OS will not be mentioned. If you do not check the box, then I will include that information in my summary of the results back to the mailing list. I will not include the names from the interoperability testing information unless I get conformation from the other implementor that it is ok to release that information, so don't worry about citing other implementations that you have tested against. I've attached a sample reply at the end... -David Borman, dab@cray.com -------- Cut here and return to dab@cray.com ----------- Name: _____________________________________________ Organization: _____________________________________ Email/Phone: ______________________________________ [ ] This information should be kept confidential I have implemented: [ ] TCP Window Scale Option [ ] TCP Timestamps Option [ ] Am using Timestamps for RTT measurements. [ ] Am using Timestamps PAWS. OS name/version: ____________________________ The code is: [ ] Experimental/research/prototype. [ ] Will be/is a Product. [ ] Not available for distribution. [ ] Will be/is available AS IS. [ ] Will be/is available for purchase. [ ] There is support for the code. Availablity Information (date available, OS version, how to get the code, who can get the code): ____________________________________________________________ ____________________________________________________________ [ ] There are known deficiencies in the implementation. Please explain. ____________________________________ ____________________________________________________ [ ] I have tested my code against other implementations. Please list: _______________________________________ ____________________________________________________ ---------- Cut here for Sample form -------------- Name: David Borman Organization: Cray Research, Inc. Email/Phone: dab@cray.com (612)683-5571 [ ] This information should be kept confidential I have implemented: [ X ] TCP Window Scale Option [ X ] TCP Timestamps Option [ X ] Am using Timestamps for RTT measurements. [ X ] Am using Timestamps PAWS. OS name/version: UNICOS 6.1 and later The code is: [ ] Experimental/research/prototype. [ X ] Will be/is a Product. [ ] Not available for distribution. [ ] Will be/is available AS IS. [ ] Will be/is available for purchase. [ X ] There is support for the code. Availablity Information (date available, OS version, how to get the code, who can get the code): The code is part of the regular UNICOS distributions, and is available today with Unicos 6.1, 7.0 and 7.C [ X ] There are known deficiencies in the implementation. The code must be enabled via an explicit setsockopt() call by the application. Just making the kernel buffers large will not do it. The code in 6.1 uses the Echo and Echo Reply options instead of the Timestamps option, and does not support PAWS. There are also mods available for 6.0 and 5.1, but these are not supported and still have bugs that are fixed in the 6.1 code. The 7.0 code supports Window Scale, Timestamps, RTT calculations and PAWS, and also has support for the Echo and Echo Reply options for backwards compatability. [ X ] I have tested my code against other implementations. Please list: Van Jacobson's code At least 3 other partial/complete implementations.   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa12720; 10 Dec 92 17:56 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa12716; 10 Dec 92 17:56 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa27605; 10 Dec 92 17:58 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.2) id AA12199; Thu, 10 Dec 92 16:58:23 CST Received: by hemlock.cray.com id AA07008; 4.1/CRI-5.6; Thu, 10 Dec 92 16:58:18 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA07004; 4.1/CRI-5.6; Thu, 10 Dec 92 16:58:14 CST Received: from zephyr.isi.edu by cray.com (4.1/CRI-MX 2.2) id AA12149; Thu, 10 Dec 92 16:58:08 CST Received: by zephyr.isi.edu (5.65c/5.61+local-9) id ; Thu, 10 Dec 1992 14:24:10 -0800 Date: Thu, 10 Dec 1992 14:24:10 -0800 Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: Bob Braden Message-Id: <199212102224.AA16199@zephyr.isi.edu> To: tcplw@cray.com, dab@berserkly.cray.com Subject: Re: 1323 implementations > > Name: _____________Bob Braden, Liming Wei_____________ > > Organization: _____Information Sciences Institute (ISI) > > Email/Phone: ___braden@isi.edu, lwei@isi.edu___________ > > [ ] This information should be kept confidential > > I have implemented: > [ X ] TCP Window Scale Option > [ X ] TCP Timestamps Option > [ X ] Am using Timestamps for RTT measurements. > [ X ] Am using Timestamps PAWS. > > OS name/version: __Sun OS 4.1____________ > The code is: > [ X ] Experimental/research/prototype. > [ ] Will be/is a Product. > [ ] Not available for distribution. > [ X ] Will be/is available AS IS. > [ ] Will be/is available for purchase. > [ ] There is support for the code. > > Availablity Information (date available, OS version, > how to get the code, who can get the code): > > ___Available Jan. 93, or when someone asks, whichever is > > ___later.__________________________________________________ > > [ ] There are known deficiencies in the implementation. > > Please explain. ____________________________________ > > ____________________________________________________ > > > [ ] I have tested my code against other implementations. > > Please list: _______________________________________ > > ____________________________________________________ >   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa09784; 18 Dec 92 2:00 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa09780; 18 Dec 92 2:00 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa15706; 18 Dec 92 2:02 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.4) id AA09432; Fri, 18 Dec 92 01:02:50 CST Received: by hemlock.cray.com id AA00547; 4.1/CRI-5.6; Fri, 18 Dec 92 01:02:46 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA00542; 4.1/CRI-5.6; Fri, 18 Dec 92 01:02:43 CST Received: from sgi.sgi.com (SGI.COM) by cray.com (4.1/CRI-MX 2.4) id AA09413; Fri, 18 Dec 92 01:02:29 CST Received: from [192.26.75.58] by sgi.sgi.com via SMTP (920330.SGI/910110.SGI) for tcplw@cray.com id AA14886; Thu, 17 Dec 92 23:02:15 -0800 Received: by rigden.wpd.sgi.com (920330.SGI/911001.SGI) for @sgi.sgi.com:tcplw@cray.com id AA19869; Thu, 17 Dec 92 23:02:14 -0800 Date: Thu, 17 Dec 92 23:02:14 -0800 Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: Rob Warnock Message-Id: <9212180702.AA19869@rigden.wpd.sgi.com> To: tcplw@cray.com Subject: Re: 50000 connections and gigabit speeds Cc: kurt@unirsvl.rsvl.unisys.com kurt@unirsvl.RSVL.UNISYS.COM writes: +--------------- | > Even in a LAN, increases in link speed give disappointingly small decreases | > in latency. Just ask anybody about NFS over Ethernet versus over FDDI... | > Gigabits are for blasting files, not transactions [unless you have 50,000 | > outstanding transactions or something].... | | By this logic I should be using 2400 baud lines instead of Ethernet or | FDDI? :-) No, seriously, am I missing something? +--------------- I think you may be. Look again at what I said before: "Even in a LAN, increases in link speed give disappointingly small decreases in latency. Just ask anybody about NFS over Ethernet versus over FDDI..." Certainly, you should consider higher speeds than 2400 baud(!), but in most LAN environments latency is a more potent inhibitor than slow link speed. Increasing your link speed only decreases the time it takes to transmit a single packet. It *doesn't* decrease the time it takes for the far end to parse, act on, and respond to that packet, and the near end to parse and act on the reply. (See below for the rest.) +--------------- | Can you only get gigabit speeds out of WANs? Why can't we build a gigabit | LAN if we can build a gigabit WAN? +--------------- Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs. That's not the point. The point is that even *LANs* are latency-limited, at gigabit speeds. +--------------- | On another note, I think that we can drop the discussion of 50,000 connections | in a transaction environment here. TCP large windows aren't going to be | helpful in that situation anyway. We are getting sidetracked on an issue that | doesn't really belong here. +--------------- But it *does* belong here! You're missing the fact that increased bandwidth *only* increases performance if the amount of data "in flight" per round-trip- time can increase, which can only happen one of three ways: 1. For a "windowed/streaming" protocol (TCP & others like it), if you increase the "window size", so you keep the faster pipe "full" while waiting for the ACKs from the remote end that *still* are going to take a while to return (since increased bandwidth doesn't help protocol handling times or transmission times for small things like ACKs). The whole point of TCP large windows is to "keep the pipe full" by allowing more than the delay-bandwidth product of data to be outstanding at once. Even on a local, fairly small FDDI ring, we have seen situations where TCP large windows were required to achieve full link rate. At gigabit speeds, it's a "must". 2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS, most RPCs), you can try increasing the *size* of each request so that at the faster link speed each request still takes about the same amount of time to send or receive. But there's a limit to how big you can make the individual RPC requests/ responses. And if the practical request/response size at the faster bit rate is much less than the round-trip latency (which can easily happen for, say, NFS on a gigabit LAN), then the throughput is dominated by the round-trip latency, and is fairly independent of the actual bitrate of the LAN (provided it's above some minimum speed, of course). For NFS, for example, experience so far puts the "knee" of the curve somewhere between Ethernet and FDDI. I know of no-one who has yet gotten more than a small fraction of an FDDI's worth of NFS traffic on a single client/server connection. 3. If you are stuck with a stop&wait request/response protocol with *really* small transactions compared to the bandwidth*delay product (e.g., typical X Windows packets even on Ethernet, or NFS over FDDI or faster), then the only way to increase total throughput is to do a *lot* of these small things in parallel, which is where the notion of "50,000 connections in a transaction environment" comes from. The latency of each individual connection is still long, and each individual request/response client/server connection's throughput is still slow, but at least you can make some use of your hundreds of megabits or your gigabits by having *lots* of simultaneously outstanding transactions. When the delay-bandwidth product is *really* large (megabytes), then trying to get link-speed performance out of a request/response transaction protocol is ludicrous (for a single client/server connection). You might as well send the entire database to the client with a windowed transport protocol (e.g., TCP), have him do a few transactions locally, and send the entire database back to the server (also with TCP or equiv). [This has been seriously considered in some environments!] Due to latency being essentially constant, there is always *some* speed above which further increases in bitrate do not substantially increase request/response transaction performance, but may (no promises!) increase a windowed/streaming protocol's performance (if the window is large enough). Do you see now? -Rob ----- Rob Warnock, MS-9U/510 rpw3@sgi.com Silicon Graphics, Inc. (415)390-1673 2011 N. Shoreline Blvd. Mountain View, CA 94043   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa01213; 18 Dec 92 3:05 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa01209; 18 Dec 92 3:05 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa02401; 18 Dec 92 3:07 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.4) id AA10040; Fri, 18 Dec 92 01:10:13 CST Received: by hemlock.cray.com id AA00749; 4.1/CRI-5.6; Fri, 18 Dec 92 01:10:05 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA00740; 4.1/CRI-5.6; Fri, 18 Dec 92 01:09:59 CST Received: from sgi.sgi.com (SGI.COM) by cray.com (4.1/CRI-MX 2.4) id AA09969; Fri, 18 Dec 92 01:09:50 CST Received: by sgi.sgi.com (920330.SGI/910110.SGI) id AA14717; Thu, 17 Dec 92 22:59:37 -0800 To: tcplw@cray.com Reply-To: Rob Warnock X-Approved: newsmail@sgi.sgi.com Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: Rob Warnock Subject: Re: 50000 connections and gigabit speeds Message-Id: Date: Fri, 18 Dec 1992 06:59:08 GMT kurt@unirsvl.RSVL.UNISYS.COM writes: +--------------- | > Even in a LAN, increases in link speed give disappointingly small decreases | > in latency. Just ask anybody about NFS over Ethernet versus over FDDI... | > | > Gigabits are for blasting files, not transactions [unless you have 50,000 | > outstanding transactions or something].... | | By this logic I should be using 2400 baud lines instead of Ethernet or | FDDI? :-) No, seriously, am I missing something? +--------------- I think you may be. Look again at what I said before: "Even in a LAN, increases in link speed give disappointingly small decreases in latency. Just ask anybody about NFS over Ethernet versus over FDDI..." Certainly, you should consider higher speeds than 2400 baud(!), but in most LAN environments latency is a more potent inhibitor than slow link speed. Increasing your link speed only decreases the time it takes to transmit a single packet. It *doesn't* decrease the time it takes for the far end to parse, act on, and respond to that packet, and the near end to parse and act on the reply. (See below for the rest.) +--------------- | Can you only get gigabit speeds out of WANs? Why can't we build a gigabit | LAN if we can build a gigabit WAN? +--------------- Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs. That's not the point. The point is that even *LANs* are latency-limited, at gigabit speeds. +--------------- | On another note, I think that we can drop the discussion of 50,000 connections | in a transaction environment here. TCP large windows aren't going to be | helpful in that situation anyway. We are getting sidetracked on an issue that | doesn't really belong here. +--------------- But it *does* belong here! You're missing the fact that increased bandwidth *only* increases performance if the amount of data "in flight" per round-trip- time can increase, which can only happen one of three ways: 1. For a "windowed/streaming" protocol (TCP & others like it), if you increase the "window size", so you keep the faster pipe "full" while waiting for the ACKs from the remote end that *still* are going to take a while to return (since increased bandwidth doesn't help protocol handling times or transmission times for small things like ACKs). The whole point of TCP large windows is to "keep the pipe full" by allowing more than the delay-bandwidth product of data to be outstanding at once. Even on a local, fairly small FDDI ring, we have seen situations where TCP large windows were required to achieve full link rate. At gigabit speeds, it's a "must". 2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS, most RPCs), you can try increasing the *size* of each request so that at the faster link speed each request still takes about the same amount of time to send or receive. But there's a limit to how big you can make the individual RPC requests/ responses. And if the practical request/response size at the faster bit rate is much less than the round-trip latency (which can easily happen for, say, NFS on a gigabit LAN), then the throughput is dominated by the round-trip latency, and is fairly independent of the actual bitrate of the LAN (provided it's above some minimum speed, of course). For NFS, for example, experience so far puts the "knee" of the curve somewhere between Ethernet and FDDI. I know of no-one who has yet gotten more than a small fraction of an FDDI's worth of NFS traffic on a single client/server connection. 3. If you are stuck with a stop&wait request/response protocol with *really* small transactions compared to the bandwidth*delay product (e.g., typical X Windows packets even on Ethernet, or NFS over FDDI or faster), then the only way to increase total throughput is to do a *lot* of these small things in parallel, which is where the notion of "50,000 connections in a transaction environment" comes from. The latency of each individual connection is still long, and each individual request/response client/server connection's throughput is still slow, but at least you can make some use of your hundreds of megabits or your gigabits by aggregating lots of sessions. When the delay-bandwidth product is *really* large (megabytes), then trying to get link-speed performance out of a request/response transaction protocol is ludicrous (for a single client/server connection). You might as well send the entire database to the client with a windowed transport protocol (e.g., TCP), have him do a few transactions locally, and send the entire database back to the server (also with TCP or equiv). [This has been seriously considered in some environments!] Due to latency being essentially constant, there is always *some* speed above which further increases in bitrate do not substantially increase request/response transaction performance, but may (no promises!) increase a windowed/streaming protocol's performance (if the window is large enough). Do you see now? -Rob ----- Rob Warnock, MS-9U/510 rpw3@sgi.com Silicon Graphics, Inc. (415)390-1673 2011 N. Shoreline Blvd. Mountain View, CA 94043   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa01223; 18 Dec 92 3:05 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa01219; 18 Dec 92 3:05 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa02406; 18 Dec 92 3:07 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.4) id AA10052; Fri, 18 Dec 92 01:10:31 CST Received: by hemlock.cray.com id AA00763; 4.1/CRI-5.6; Fri, 18 Dec 92 01:10:10 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA00756; 4.1/CRI-5.6; Fri, 18 Dec 92 01:10:07 CST Received: from sgi.sgi.com (SGI.COM) by cray.com (4.1/CRI-MX 2.4) id AA09998; Fri, 18 Dec 92 01:09:58 CST Received: by sgi.sgi.com (920330.SGI/910110.SGI) id AA15435; Thu, 17 Dec 92 23:08:20 -0800 To: tcplw@cray.com Reply-To: Rob Warnock X-Approved: newsmail@sgi.sgi.com Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: Rob Warnock Subject: Re: 50000 connections and gigabit speeds Message-Id: Date: Fri, 18 Dec 1992 07:08:14 GMT kurt@unirsvl.RSVL.UNISYS.COM writes: +--------------- | > Even in a LAN, increases in link speed give disappointingly small decreases | > in latency. Just ask anybody about NFS over Ethernet versus over FDDI... | > Gigabits are for blasting files, not transactions [unless you have 50,000 | > outstanding transactions or something].... | | By this logic I should be using 2400 baud lines instead of Ethernet or | FDDI? :-) No, seriously, am I missing something? +--------------- I think you may be. Look again at what I said before: "Even in a LAN, increases in link speed give disappointingly small decreases in latency. Just ask anybody about NFS over Ethernet versus over FDDI..." Certainly, you should consider higher speeds than 2400 baud(!), but in most LAN environments latency is a more potent inhibitor than slow link speed. Increasing your link speed only decreases the time it takes to transmit a single packet. It *doesn't* decrease the time it takes for the far end to parse, act on, and respond to that packet, and the near end to parse and act on the reply. (See below for the rest.) +--------------- | Can you only get gigabit speeds out of WANs? Why can't we build a gigabit | LAN if we can build a gigabit WAN? +--------------- Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs. That's not the point. The point is that even *LANs* are latency-limited, at gigabit speeds. +--------------- | On another note, I think that we can drop the discussion of 50,000 connections | in a transaction environment here. TCP large windows aren't going to be | helpful in that situation anyway. We are getting sidetracked on an issue that | doesn't really belong here. +--------------- But it *does* belong here! You're missing the fact that increased bandwidth *only* increases performance if the amount of data "in flight" per round-trip- time can increase, which can only happen one of three ways: 1. For a "windowed/streaming" protocol (TCP & others like it), if you increase the "window size", so you keep the faster pipe "full" while waiting for the ACKs from the remote end that *still* are going to take a while to return (since increased bandwidth doesn't help protocol handling times or transmission times for small things like ACKs). The whole point of TCP large windows is to "keep the pipe full" by allowing more than the delay-bandwidth product of data to be outstanding at once. Even on a local, fairly small FDDI ring, we have seen situations where TCP large windows were required to achieve full link rate. At gigabit speeds, it's a "must". 2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS, most RPCs), you can try increasing the *size* of each request so that at the faster link speed each request still takes about the same amount of time to send or receive. But there's a limit to how big you can make the individual RPC requests/ responses. And if the practical request/response size at the faster bit rate is much less than the round-trip latency (which can easily happen for, say, NFS on a gigabit LAN), then the throughput is dominated by the round-trip latency, and is fairly independent of the actual bitrate of the LAN (provided it's above some minimum speed, of course). For NFS, for example, experience so far puts the "knee" of the curve somewhere between Ethernet and FDDI. I know of no-one who has yet gotten more than a small fraction of an FDDI's worth of NFS traffic on a single client/server connection. 3. If you are stuck with a stop&wait request/response protocol with *really* small transactions compared to the bandwidth*delay product (e.g., typical X Windows packets even on Ethernet, or NFS over FDDI or faster), then the only way to increase total throughput is to do a *lot* of these small things in parallel, which is where the notion of "50,000 connections in a transaction environment" comes from. The latency of each individual connection is still long, and each individual request/response client/server connection's throughput is still slow, but at least you can make some use of your hundreds of megabits or your gigabits by having *lots* of simultaneously outstanding transactions. When the delay-bandwidth product is *really* large (megabytes), then trying to get link-speed performance out of a request/response transaction protocol is ludicrous (for a single client/server connection). You might as well send the entire database to the client with a windowed transport protocol (e.g., TCP), have him do a few transactions locally, and send the entire database back to the server (also with TCP or equiv). [This has been seriously considered in some environments!] Due to latency being essentially constant, there is always *some* speed above which further increases in bitrate do not substantially increase request/response transaction performance, but may (no promises!) increase a windowed/streaming protocol's performance (if the window is large enough). Do you see now? -Rob ----- Rob Warnock, MS-9U/510 rpw3@sgi.com Silicon Graphics, Inc. (415)390-1673 2011 N. Shoreline Blvd. Mountain View, CA 94043   Received: from ietf.nri.reston.va.us by IETF.CNRI.Reston.VA.US id aa05309; 18 Dec 92 10:17 EST Received: from CNRI.RESTON.VA.US by IETF.CNRI.Reston.VA.US id aa05305; 18 Dec 92 10:17 EST Received: from timbuk.cray.com by CNRI.Reston.VA.US id aa12653; 18 Dec 92 10:19 EST Received: from hemlock.cray.com by cray.com (4.1/CRI-MX 2.4) id AA15789; Fri, 18 Dec 92 09:19:30 CST Received: by hemlock.cray.com id AA15752; 4.1/CRI-5.6; Fri, 18 Dec 92 09:19:28 CST Received: from cray.com (timbuk.cray.com) by hemlock.cray.com id AA15748; 4.1/CRI-5.6; Fri, 18 Dec 92 09:19:24 CST Received: from relay1.UU.NET by cray.com (4.1/CRI-MX 2.4) id AA15786; Fri, 18 Dec 92 09:19:22 CST Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay1.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA18640; Fri, 18 Dec 92 10:19:20 -0500 Received: from unirsvl.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 101831.17294; Fri, 18 Dec 1992 10:18:31 EST Received: by unirsvl.RSVL.UNISYS.COM (smail2.5) id AA02092; 18 Dec 92 08:58:00 CST (Fri) Subject: Re: 50000 connections and gigabit speeds To: Rob Warnock Date: Fri, 18 Dec 92 8:57:57 CST Cc: tcplw@cray.com In-Reply-To: <9212180702.AA19869@rigden.wpd.sgi.com>; from "Rob Warnock" at Dec 17, 92 11:02 pm X-Mailer: ELM [version 2.2 PL0] Message-Id: <9212180857.AA02088@unirsvl.RSVL.UNISYS.COM> Sender:ietf-archive-request@IETF.CNRI.Reston.VA.US From: kurt@unirsvl.rsvl.unisys.com > > Do you see now? > > > -Rob > Yes I do. Thanks for the info, it pointed out a few things that I had not thought of. Kurt