Subject: Re: [ledbat] list of reasons for needing multiple TCP connections

From: "Robb Topolski" <[email-masked]>
Date: Mon, 1 Dec 2008 14:38:36 -0800

Most home "routers" have all been modified by now not to crash when
> BitTorrent has opened thousands of TCP connections. These connections
> consume mapping table resources and were a problem until it they were
> garbage-collected.
>
>
I claim to have no handle on what "most" home routers do, but I doubt this
assertion.

NAT table size continues to be a problem with NAT limit, and not because
BitTorrent typically opens thousands of TCP connections (it doesn't, nor do
trackers generally provide the thousands of peer addresses necessary to go
with those connections), but because the combination of the UDP-based DHT in
addition to 100 TCP connections and a decent amount of WAN-LAN traffic is
enough to overcome NAT tables in popular routers being sold today.

http://www.smallnetbuilder.com/component/option,com_chart/Itemid,189/chart,124/
(note 200 is the upper limit of their test method, but of use to this
message is the number and market position of routers that support less than
200)

Belkin's F5D8232-4 is 41st in Amazon's sales rank for "routers" and maxed
out at 180 connections.
Netgear FVS124G is 14th in "gigabit routers" and maxes out at 196.
Apple's BL053LL/A is 1st in "gigabit routers" and maxes out at 128.

Do they crash?  Beats me.  But NAT table size limit is definitely a
consideration.  The behaviors when those limits are reached are varied and,
even on some recently sold equipment, sometimes less than graceful. One of
the most common failure modes I've run across is that DNS Relay stops
working or the configuration pages stop responding while pre-existing
connections continue uninterrupted!  (I'm guessing that the daemons
supporting these functions either get killed or lack enough memory to do
anything, they themselves are shut out of the NAT table, or maybe they won't
launch when CPU load is greater than some amount.)

I think it continues to deserve inclusion.  If someone has data to the
contrary, please bring it.

-- 
Robb Topolski ([email-masked])
Hillsboro, Oregon USA
http://www.funchords.com/
_______________________________________________
ledbat mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ledbat

---

From: Richard Bennett <[email-masked]>
Date: Mon, 01 Dec 2008 15:00:09 -0800

Au contraire, Robb, in the course of a 24 hour period you will typically 
see a total of well over 4000 NAT table entries for BitTorrent when it's 
in use. The fact that BitTorrent doesn't use all the thousands at the 
same time isn't the issue, it's the fact that a mapping entry has to be 
made in the table each time a packet is sent from an end system behind 
the NAT to one in front of the NAT. TCP is actually more friendly to 
this problem than UDP, because it give the NAT a clue about when it's 
safe to free an entry. Stateless  UDP is much more of a problem.

RB

Robb Topolski wrote:
>
>
>     Most home "routers" have all been modified by now not to crash
>     when BitTorrent has opened thousands of TCP connections. These
>     connections consume mapping table resources and were a problem
>     until it they were garbage-collected.
>
>
> I claim to have no handle on what "most" home routers do, but I doubt 
> this assertion.
>
> NAT table size continues to be a problem with NAT limit, and not 
> because BitTorrent typically opens thousands of TCP connections (it 
> doesn't, nor do trackers generally provide the thousands of peer 
> addresses necessary to go with those connections), but because the 
> combination of the UDP-based DHT in addition to 100 TCP connections 
> and a decent amount of WAN-LAN traffic is enough to overcome NAT 
> tables in popular routers being sold today. 
>
> message is the number and market position of routers that support less 
> than 200)
>
> Belkin's F5D8232-4 is 41st in Amazon's sales rank for "routers" and 
> maxed out at 180 connections.
> consideration.  The behaviors when those limits are reached are varied 
> and, even on some recently sold equipment, sometimes less than 
> graceful. One of the most common failure modes I've run across is that 
> DNS Relay stops working or the configuration pages stop responding 
> while pre-existing connections continue uninterrupted!  (I'm guessing 
> that the daemons supporting these functions either get killed or lack 
> enough memory to do anything, they themselves are shut out of the NAT 
> table, or maybe they won't launch when CPU load is greater than some 
> amount.) 
>
> Robb Topolski ([email-masked] <mailto:[email-masked]>)
> Hillsboro, Oregon USA
> ------------------------------------------------------------------------
>

-- 
Richard Bennett

---

From: "Robb Topolski" <[email-masked]>
Date: Mon, 1 Dec 2008 15:27:33 -0800

>
> in the course of a 24 hour period you will typically see a total of well
> over 4000 NAT table entries for BitTorrent when it's in use.

Good grief.

"In the course of a 24-hour period" is a huge qualifier, not to mention
irrelevant.  How big is the NAT table over the course of a year?   You can
really demonize P2P with a stat like that!!

The only relevant NAT table is the one you have, now.

TCP is actually more friendly to this problem than UDP, because it give the
> NAT a clue about when it's safe to free an entry.
>

I don't know that TCP is any more or less friendly.  Sure, it gives state,
but you have to track that state and do an orphan timer.  Meanwhile, a lot
of TCP apps send keepalives anyway, rendering all that pretty much
meaningless.

Stateless  UDP is much more of a problem.

With UDP, it's much less of a problem.  Devices can just expire the entry
after sufficient time has passed since the last traversal -- which is what
they already do.

With TCP entries, owing to the chance that keepalives aren't used, my
favorite router sets a 2.5 hour timer.  With UDP entries, it's 5 minutes.

On Mon, Dec 1, 2008 at 3:00 PM, Richard Bennett <[email-masked]> wrote:

> see a total of well over 4000 NAT table entries for BitTorrent when it's in
> use. The fact that BitTorrent doesn't use all the thousands at the same time
> isn't the issue, it's the fact that a mapping entry has to be made in the
> table each time a packet is sent from an end system behind the NAT to one in
> front of the NAT. TCP is actually more friendly to this problem than UDP,
> because it give the NAT a clue about when it's safe to free an entry.
> Stateless  UDP is much more of a problem.
>
> RB
>
> Robb Topolski wrote:
>
>
>
> ------------------------------
>
> ledbat mailing [email-masked]://www.ietf.org/mailman/listinfo/ledbat
>
>
> --
> Richard Bennett
>
>

---

From: Richard Bennett <[email-masked]>
Date: Mon, 01 Dec 2008 15:35:16 -0800

Setting an expiration period of 5 minutes for UDP entries in a NAT table 
will certainly kill UTP. Is that what you really want, Robb? The mapping 
entries have to be kept around as long as there's any likelihood of 
traffic coming in for them, which can be a very long time for P2P.

A little more tech and a little less snark would actually be beneficial.

RB

Robb Topolski wrote:
>
>     in the course of a 24 hour period you will typically see a total
>     of well over 4000 NAT table entries for BitTorrent when it's in use.
>
>
> Good grief. 
>
> "In the course of a 24-hour period" is a huge qualifier, not to 
> mention irrelevant.  How big is the NAT table over the course of a 
> year?   You can really demonize P2P with a stat like that!! 
>
>     TCP is actually more friendly to this problem than UDP, because it
>     give the NAT a clue about when it's safe to free an entry.
>
>
> I don't know that TCP is any more or less friendly.  Sure, it gives 
> state, but you have to track that state and do an orphan timer.  
> Meanwhile, a lot of TCP apps send keepalives anyway, rendering all 
> that pretty much meaningless. 
>
> With UDP, it's much less of a problem.  Devices can just expire the 
> entry after sufficient time has passed since the last traversal -- 
> which is what they already do.
>
> On Mon, Dec 1, 2008 at 3:00 PM, Richard Bennett <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Au contraire, Robb, in the course of a 24 hour period you will
>     typically see a total of well over 4000 NAT table entries for
>     BitTorrent when it's in use. The fact that BitTorrent doesn't use
>     all the thousands at the same time isn't the issue, it's the fact
>     that a mapping entry has to be made in the table each time a
>     packet is sent from an end system behind the NAT to one in front
>     of the NAT. TCP is actually more friendly to this problem than
>     UDP, because it give the NAT a clue about when it's safe to free
>     an entry. Stateless  UDP is much more of a problem.
>
>     RB
>
>     Robb Topolski wrote:
>>
>>
>>         Most home "routers" have all been modified by now not to
>>         crash when BitTorrent has opened thousands of TCP
>>         connections. These connections consume mapping table
>>         resources and were a problem until it they were
>>         garbage-collected.
>>
>>
>>     I claim to have no handle on what "most" home routers do, but I
>>     doubt this assertion.
>>
>>     because BitTorrent typically opens thousands of TCP connections
>>     (it doesn't, nor do trackers generally provide the thousands of
>>     peer addresses necessary to go with those connections), but
>>     because the combination of the UDP-based DHT in addition to 100
>>     TCP connections and a decent amount of WAN-LAN traffic is enough
>>     to overcome NAT tables in popular routers being sold today. 
>>
>>     (note 200 is the upper limit of their test method, but of use to
>>     this message is the number and market position of routers that
>>     support less than 200)
>>
>>     Belkin's F5D8232-4 is 41st in Amazon's sales rank for "routers"
>>     and maxed out at 180 connections.
>>     Do they crash?  Beats me.  But NAT table size limit is definitely
>>     a consideration.  The behaviors when those limits are reached are
>>     varied and, even on some recently sold equipment, sometimes less
>>     than graceful. One of the most common failure modes I've run
>>     across is that DNS Relay stops working or the configuration pages
>>     stop responding while pre-existing connections continue
>>     uninterrupted!  (I'm guessing that the daemons supporting these
>>     functions either get killed or lack enough memory to do anything,
>>     they themselves are shut out of the NAT table, or maybe they
>>     won't launch when CPU load is greater than some amount.) 
>>
>>     I think it continues to deserve inclusion.  If someone has data
>>     to the contrary, please bring it.
>>
>>
>>     -- 
>>     _______________________________________________ ledbat mailing
>>     list [email-masked] <mailto:[email-masked]>

-- 
Richard Bennett

---

From: "Robb Topolski" <[email-masked]>
Date: Mon, 1 Dec 2008 18:46:57 -0800

> will certainly kill UTP. Is that what you really want, Robb?
>

Don't kill the messenger, Richard.  I'd love to tell you that it takes 180
seconds, but it simply doesn't. Whoever designed it decides It takes 300.

The mapping entries have to be kept around as long as there's any likelihood
> of traffic coming in for them, which can be a very long time for P2P.

How long, Richard?

Was that for me?  Or, were are you quoting today's reviews of your article?

http://gigaom.com/2008/12/01/bittorrent-at-war-with-voip-hardly/

http://torrentfreak.com/will-utorrent-really-kill-the-internet-081201/

http://www.thestandard.com/news/2008/12/01/bittorrent-register-report-protocol-change-utter-nonsense

http://icrontic.com/articles/will-bittorrents-switch-to-udp-kill-the-internet

http://www.dslreports.com/shownews/UDP-BitTorrent-Will-Destroy-The-Interwebs-99400

Robb

On Mon, Dec 1, 2008 at 3:35 PM, Richard Bennett <[email-masked]> wrote:

> entries have to be kept around as long as there's any likelihood of traffic
> coming in for them, which can be a very long time for P2P.
>

---

From: Richard Bennett <[email-masked]>
Date: Mon, 01 Dec 2008 19:06:42 -0800

Actually, this article in DSL Reports started the recent discussion 
about BitTorrent over UDP:

http://www.dslreports.com/shownews/New-UDP-uTorrent-Takes-Aim-At-Throttling-99366

The title is: "New UDP uTorrent Takes Aim At Throttling". The author, 
Karl Bode, is aware that UDP is one of the methods that P2P programmers 
have discussed over the years for avoiding traffic shaping. There's 
nothing particularly obscene about attempting to work through the 
implications of trade-offs; this is what engineers do.

Several people have objected to the implication that the sole purpose of 
making UDP the default transport was management evasion, and I'll 
correct that notion in a follow-up piece. That being said, there are 
still important issues to explore in relation to the Internet 
infrastructure, for "whoever designs it."

RB

Robb Topolski wrote:
>
>     Setting an expiration period of 5 minutes for UDP entries in a NAT
>     table will certainly kill UTP. Is that what you really want, Robb?
>
>
> Don't kill the messenger, Richard.  I'd love to tell you that it takes 
> 180 seconds, but it simply doesn't. Whoever designed it decides It 
> takes 300. 
>
>     The mapping entries have to be kept around as long as there's any
>     likelihood of traffic coming in for them, which can be a very long
>     time for P2P.
>
>
> How long, Richard? 
>
> On Mon, Dec 1, 2008 at 3:35 PM, Richard Bennett <[email-masked] 
>     Setting an expiration period of 5 minutes for UDP entries in a NAT
>     table will certainly kill UTP. Is that what you really want, Robb?
>     The mapping entries have to be kept around as long as there's any
>     likelihood of traffic coming in for them, which can be a very long
>     time for P2P.
>

-- 
Richard Bennett

---

From: Nicholas Weaver <[email-masked]>
Date: Mon, 1 Dec 2008 19:10:01 -0800

On Dec 1, 2008, at 7:06 PM, Richard Bennett wrote:

> The title is: "New UDP uTorrent Takes Aim At Throttling". The  
> author, Karl Bode, is aware that UDP is one of the methods that P2P  
> programmers have discussed over the years for avoiding traffic  
> shaping. There's nothing particularly obscene about attempting to  
> work through the implications of trade-offs; this is what engineers  
> do.
>
> Several people have objected to the implication that the sole  
> purpose of making UDP the default transport was management evasion,  
> and I'll correct that notion in a follow-up piece. That being said,  
> there are still important issues to explore in relation to the  
> Internet infrastructure, for "whoever designs it."

We've had confirmation that the udp protocol being on-by-default in  
uTorrent is the same AIMD + delay-based that was presented at the IETF.

AIMD + delay-based means it can't make the ISP's congestion problem  
worse (it may make it better but thats an open question), but it  
definatly makes the access-link congestion better.

So rather than "getting around" traffic management (which doesn't work  
anyway), this is specifically about reducing the network impact,  
especially on the local access point.

---

From: Reinaldo Penno <[email-masked]>
Date: Mon, 1 Dec 2008 17:29:06 -0800

Hello,

Any NAT developer has a bag of tricks to deal with state shortage. He/she
can:

1 Drop any new sessions/conn
2 Nuke the oldest one
3 Nuke the mostly idle one
4 Combination of 2 & 3
5 Use tickle
6 Set up timers depending on application ports
7 Use ALGs to track end of dialog
8 Smart nuke of a per application basis (e.g. Safe to nuke IM, but only nuke
business VoIP control as last measure)

I could go on.

The strategy chosen depends on availability of ALGs, configurable timers,
home routers vs. access vs. something else, etc.

As far as TCP (not application) keep alive goes, it is 2 hours. A lot of
stuff can happen in two hours.

Thanks,

Reinaldo

On 12/1/08 3:35 PM, "Richard Bennett" <[email-masked]> wrote:

> Setting an expiration period of 5 minutes for UDP entries in a NAT table will
> certainly kill UTP. Is that what you really want, Robb? The mapping entries
> have to be kept around as long as there's any likelihood of traffic coming in
> for them, which can be a very long time for P2P.
> 
>> but you have to track that state and do an orphan timer. Meanwhile, a lot of
>> TCP apps send keepalives anyway, rendering all that pretty much meaningless.
>>  
>>  
>>> Au contraire, Robb, in the course of a 24 hour period you will typically see
>>> a total of well over 4000 NAT table entries for BitTorrent when it's in use.
>>> The fact that BitTorrent doesn't use all the thousands at the same time
>>>> with those connections), but because the combination of the UDP-based DHT
>>>> in addition to 100 TCP connections and a decent amount of WAN-LAN traffic
>>>> is enough to overcome NAT tables in popular routers being sold today.
>>>>  
>>>>  
>>>> anything, they themselves are shut out of the NAT table, or maybe they

---

From: Eric Spaeth <[email-masked]>
Date: Mon, 01 Dec 2008 22:09:10 -0600

Robb Topolski wrote:
> 2.  (Until I'm blue in the face, I swear), P2P developers generally 
> are fans of the Internet and are unlikely to intentionally develop in 
> ways harmful to the network.  I don't know how else to day it.  They 
> don't ##### where they live.  They surf websites and take VOIP calls, 
> too. 
The problem with this statement is that P2P troubles are an n-user 
problem; it is often difficult for people to discern impact without 
seeing a direct and immediate negative effect from their actions.  Even 
so, I tend to think that most users don't care about the impact they 
cause on the network as long as they get theirs.  You'll notice that 
every time the topic of ISP throttling comes up on a public forum the 
first question is almost always "How do I get around it?"  with very 
little focus on "Why would the ISP spend the time and money to implement 
a throttle for P2P?"

I think BitTorrent's creator Bram Cohen said it best:  .... it's 
something he [Bram Cohen] predicted when he first thought up BitTorrent. 
"My whole idea was, 'Let's use up a lot of bandwidth,'" he laughs. "I 
had a friend who said, 'Well, ISPs won't like that.' And I said, 'Why 
should I care?'"
Source: 
http://www.sfweekly.com/2008-01-23/news/bittorrent-comcast-eff-antipathetic-to-fcc-regulation-of-p2p-traffic/3

>     It's clear that the users of BitTorrent see this as simply another
>     means of grabbing more network resources.
>
>
> 3.  I'm not agreeing with that conclusion, as it's just as plausible 
> that they want to reduce their own support load; a lot of effort is 
> dedicated to teaching users how to avoid overrunning the buffers in 
> their modem.  However, if "more network resources" are they're there 
> to "grab" and doing so doesn't interfere with anybody, then so what?  
> The design of TCP is to keep trying to grab more network resources, 
> and most applications do nothing to change that behavior.
TCP is a tamable beast; we've had decades to work out issues of 
congestion-avoidance synchronization, figure out different algorithms 
for window scaling, and implement solutions like RED to eliminate or 
reduce tail-drop issues.  Instead we're talking about replacing TCP 
transport with a hastily crafted replacement using UDP underpinnings 
primarily to get around ISP interference.  I spend quite a bit of my 
time digging through packet captures, troubleshooting application issues 
caused by developers who suffer from chronic clue deficiency when it 
comes to network awareness; (ie, it worked great from my desk to the dev 
server in the same building, but when we deployed it to users in India 
the application is slow.  Can you fix the network?)  I have very little 
faith that this will get anywhere close to right on the first attempt. 

I'm really wondering what impact this is going to have on routing at the 
backbone layer.   Many large network operators are using intelligent 
route reflectors like the Internap FCP platform or the 
Avaya/RouteScience appliances to attempt to route around congested 
links.  One of the ways they make path selection decisions is to analyze 
TCP flow performance (look for retransmits) or read router NetFlow data 
to evaluate performance of various paths.   How will the use of uTP 
affect route selection by these appliances once they start to get a 
significant amount of traffic for which they are unable to ascertain 
flow statistics?

-Eric

---

From: "Robb Topolski" <[email-masked]>
Date: Tue, 2 Dec 2008 00:16:48 -0800

On Mon, Dec 1, 2008 at 8:09 PM, Eric Spaeth <[email-masked]> wrote:

> Robb Topolski wrote:
>
>> 2.  (Until I'm blue in the face, I swear), P2P developers generally are
>> fans of the Internet and are unlikely to intentionally develop in ways
>> harmful to the network.  I don't know how else to day it.  They don't #####
>> where they live.  They surf websites and take VOIP calls, too.
>>
> The problem with this statement is that P2P troubles are an n-user problem;
> it is often difficult for people to discern impact without seeing a direct
> and immediate negative effect from their actions.  Even so, I tend to think
> that most users don't care about the impact they cause on the network as
> long as they get theirs.  You'll notice that every time the topic of ISP
> throttling comes up on a public forum the first question is almost always
> "How do I get around it?"  with very little focus on "Why would the ISP
> spend the time and money to implement a throttle for P2P?"
>
> I think BitTorrent's creator Bram Cohen said it best:  .... it's something
> he [Bram Cohen] predicted when he first thought up BitTorrent. "My whole
> idea was, 'Let's use up a lot of bandwidth,'" he laughs. "I had a friend who
> said, 'Well, ISPs won't like that.' And I said, 'Why should I care?'"
> Source:
>> 3.  I'm not agreeing with that conclusion, as it's just as plausible that
>> they want to reduce their own support load; a lot of effort is dedicated to
>> teaching users how to avoid overrunning the buffers in their modem.
>>  However, if "more network resources" are they're there to "grab" and doing
>> so doesn't interfere with anybody, then so what?  The design of TCP is to
>> keep trying to grab more network resources, and most applications do nothing
>> to change that behavior.
>>
> congestion-avoidance synchronization, figure out different algorithms for
> window scaling, and implement solutions like RED to eliminate or reduce
> tail-drop issues.  Instead we're talking about replacing TCP transport with
> a hastily crafted replacement

>From where do you get the word, "hastily?"

> using UDP underpinnings primarily to get around ISP interference.  I

> I have very little faith that this will get anywhere close to right on the
> first attempt.

Experience tells me that you're right, and it is also tells me that failure
is no reason not to try.

> backbone layer.   Many large network operators are using intelligent route
> reflectors like the Internap FCP platform or the Avaya/RouteScience
> appliances to attempt to route around congested links.  One of the ways they
> make path selection decisions is to analyze TCP flow performance (look for
> retransmits) or read router NetFlow data to evaluate performance of various
> paths.

Won't that still work, anyway?  Won't any reduction in clues be
porportionate?  Still the IP network has to interoperate in the basic way.
We shoudln't enable session-monitoring, session-managing, or
application-layer DPI to limit innovation of lighter-than-TCP protocols.

Call it a bail-out. :-)  DPI-based hardware vendors will have something new
to sell in 6 months and past-purchasers will have some tough questions about
why they can't shove all the BitTorrent traffic down that 10% sized virtual
route anymore. Meanwhile the bottom-up flexibility intended for development
on IP-based protocols is preserved -- and the part I love, perhaps we'll get
to see whose routing or dropping UDP packets differently than expected.
Sunshine is a great disinfectant.

What's also great about this is the folks that do care about congestion are
endeavoring to do this right.  This isn't a hackish attempt to ignore
signals like latency and drops in order to gain pipe-share over the
neighbors, it's an attempt to treat traffic in ways that -- in the end --
give preference to actual foreground network uses over their own background
uses.  It's conservative.  It's disclosed.  It's going to be released
openly.  The folks doing it are participating in the open.

That's simply awesome, it's the role-model behavior we out to expect out of
everyone providing services and equipment for the Internet.

Robb Topolski

---

From: "Robb Topolski" <[email-masked]>
Date: Tue, 2 Dec 2008 00:17:25 -0800

> business VoIP control as last measure)
Hey Reinaldo!

Yeah, I know (except for #5, I don't know what that is).  Still seems like a
lot of them seem to choose #19 - Slowly Grind to a mystifying Halt or #14 -
Reboot without explanation.  (Although the reboot is probably a watchdog
function reacting to the grinding halt!)

  :-)

PS:  Is it possible that some devs just don't know what to do when the table
is full or nearly full?  I know that these guys are working on SDKs from a
small handful of vendors, but I've never actually seen one or the
documentation that comes with it.