Subject: [Stackevo-discuss] Fwd: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt

From: Michael Welzl <[email-masked]>
Date: Wed, 23 Mar 2016 12:42:18 +0100

Hi all,

I'm sending this here as well because it relates to the idea of "running the Internet over UDP" (which, I noticed, is presentation that Brian will give in MAPRG: https://datatracker.ietf.org/meeting/95/agenda/maprg/ ).
This draft explains how we think it should be done with TCP; we believe that it can have quite significant positive side-effects, mainly because of the opportunity to combine the congestion controls of multiple connections.

The draft also briefly discusses the issue of inserting an extra header (such as SPUD) in between the UDP and encapsulated TCP header.
https://tools.ietf.org/html/draft-trammell-spud-req-02#section-8.3 raises some concerns about such in-band SPUD signalling. What we found here is that this is quite unproblematic in case of TCP when inserted by the sender (it reduces the MSS and must be taken into account when calculating space on the sender side, but this really isn't a big problem).

Further discussion related to this draft should happen on the ICCRG list, please.

Cheers,
Michael

> Begin forwarded message:
> 
> From: Michael Welzl <[email-masked]>
> Subject: Fwd: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
> Date: 23 Mar 2016 09:04:48 CET
> To: tcpm IETF list <[email-masked]>, [email-masked]
> 
> Hi,
> 
> This could be of interest to folks here. We'd be glad to get your comments - please send them to the ICCRG list only, at [email-masked].
> 
> Cheers,
> Michael
> 
> 
>> Begin forwarded message:
>> 
>> From: Michael Welzl <[email-masked]>
>> Subject: Fwd: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
>> Date: 23 Mar 2016 09:02:38 CET
>> To: [email-masked]
>> 
>> (..)
>> 
>> Chair hat off:
>> 
>> Dear all,
>> 
>> I'm submitting this to ICCRG as the primary (and possibly ultimate? We'll see) forum to discuss this draft. We'd be happy to get your comments - please keep them on the ICCRG list.
>> We have FreeBSD code, from which we can already confirm that the encapsulation is indeed relatively easy to do (as our draft states). However, it doesn't do the congestion control coupling yet; we're working on it. For that, so far, we have simulations - from these, we hope to be able to show some first results in BA.
>> 
>> Cheers,
>> Michael
>> 
>> 
>> 
>>> Begin forwarded message:
>>> 
>>> From: <[email-masked]>
>>> Subject: New Version Notification for draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
>>> Date: 21 Mar 2016 18:45:57 CET
>>> To: Jianjie You <[email-masked]>, Michael Welzl <[email-masked]>, Safiqul Islam <[email-masked]>, Kristian Hiorth <[email-masked]>
>>> Resent-From: <[email-masked]>
>>> 
>>> 
>>> A new version of I-D, draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
>>> has been successfully submitted by Michael Welzl and posted to the
>>> IETF repository.
>>> 
>>> Name:		draft-welzl-irtf-iccrg-tcp-in-udp
>>> Revision:	00
>>> Title:		TCP in UDP
>>> Document date:	2016-03-21
>>> Group:		Individual Submission
>>> Pages:		17
>>> URL:            https://www.ietf.org/internet-drafts/draft-welzl-irtf-iccrg-tcp-in-udp-00.txt
>>> Status:         https://datatracker.ietf.org/doc/draft-welzl-irtf-iccrg-tcp-in-udp/
>>> Htmlized:       https://tools.ietf.org/html/draft-welzl-irtf-iccrg-tcp-in-udp-00
>>> 
>>> 
>>> Abstract:
>>> This document specifies a method to encapsulate multiple TCP
>>> connections using only one UDP port number pair.  Doing so allows for
>>> a relatively easy implementation of coupled congestion control for
>>> the TCP connections.  This can have several performance benefits, and
>>> it makes it possible to precisely assign a share of the congestion
>>> window to the connections based on priorities.  It also enables use
>>> of UDP-based NAT traversal techniques, and it can act as a framework
>>> for experimentation with novel changes to the TCP standard.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Please note that it may take a couple of minutes from the time of submission
>>> until the htmlized version and diff are available at tools.ietf.org.
>>> 
>>> The IETF Secretariat
>>> 
>>

---

From: Joe Touch <[email-masked]>
Date: Wed, 23 Mar 2016 16:05:15 -0700

> This draft explains how we think it should be done with TCP; we believe that it can have quite significant positive side-effects, mainly because of the opportunity to combine the congestion control…

Ick.

We went through this before multiple times. One notable was the impact
of chunking and muxing in HTTP, effectively reinventing IP inside.

"A person with one watch always knows what time it is. A person with two
is never sure."

I can't imagine what combined congestion control of TCP would mean
inside the network - we had a hard enough time defining it for either
TCB sharing or the Congestion Manager just at the endpoints.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Thu, 24 Mar 2016 00:43:07 +0100

> On 24. mar. 2016, at 00.05, Joe Touch <[email-masked]> wrote:
> 
> 
> 
> is never sure.”

I don’t think I follow this, sorry.

> TCB sharing or the Congestion Manager just at the endpoints.

I don’t get this either; it sounds like a misunderstanding?

So the encapsulation here is just meant to force packets to take the same path - else the Congestion Manager may be doing something pretty wrong.
The sender-side behavior then couples connections; this could be done via the Congestion Manager, or in a way similar to TCB sharing, or with our algorithm (which is somewhat similar to TCB sharing). That’s why we call it “example algorithm” - think of this as “UDP encapsulation forces TCP connections on the same path so we can use the CM”. That’s really all it is.

Cheers,
Michael

---

From: Tom Herbert <[email-masked]>
Date: Wed, 23 Mar 2016 16:51:28 -0700

> The sender-side behavior then couples connections; this could be done via the Congestion Manager, or in a way similar to TCB sharing, or with our algorithm (which is somewhat similar to TCB sharing…
You should be able to accomplish the "same path" behavior by setting
IPv6 flow label same value for different connections between same IP
pair. This avoids all the unpleasantness that accompanies
encapsulation.

Tom

> Cheers,
> Michael
>
> _______________________________________________
> Stackevo-discuss mailing list
> [email-masked]
> https://www.iab.org/mailman/listinfo/stackevo-discuss

---

From: Joe Touch <[email-masked]>
Date: Wed, 23 Mar 2016 16:52:39 -0700

> I don’t think I follow this, sorry.

Having one uniform layer at which multiplexing occurs makes things easy.
Having two means you have to wonder what the other is doing all the time
and whether the two correlate.

> So the encapsulation here is just meant to force packets to take the 
> same path - else the Congestion Manager may be doing something
> pretty wrong.

The CM is supposed to react to E2E congestion, which is the sum of all
effects of the different paths that packets can take.

The idea that packets need to take a single path is an ATM concept, not
IP. I think the Internet would be a lot better off if we just admit that
it's *based* on the idea that packets can use different paths, and that
this is a good thing.

> The sender-side behavior then couples connections; this could be 
> done via the Congestion Manager, or in a way similar to TCB sharing,
> or with our algorithm (which is somewhat similar to TCB sharing).
> That’s why we call it “example algorithm” - think of this as “UDP
> encapsulation forces TCP connections on the same path so we can use
> the CM”. That’s really all it is.

"all it is" is a complete reimplementation of everything in IP inside
UDP -- because everything that IP provides will ultimately be need by
this layer, as well as a way to map this to the corresponding IP mechanism.

I think it would be useful to take a deeper look at the implications
before jumping into this water.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 23 Mar 2016 16:57:01 -0700

On 3/23/2016 4:51 PM, Tom Herbert wrote:
...
> pair. This avoids all the unpleasantness that accompanies

You would need to do IP encapsulation AND use a flow label on that
encaps. Otherwise, you'd have different IP addresses with the same flow
label - which aren't expected to do anything useful.

See RFC6437, Sec. 2.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Wed, 23 Mar 2016 17:51:35 -0700

> label - which aren't expected to do anything useful.
Oh, I guess I'm missing something. If the (destination) addresses are
different how does encapsulating in UDP allow packets for these
connections to take the same path?

Tom

---

From: Michael Welzl <[email-masked]>
Date: Thu, 24 Mar 2016 07:43:07 +0100

> On 24. mar. 2016, at 01.51, Tom Herbert <[email-masked]> wrote:
> 

Sounds good to me! Generally, our intention is to enable congestion control coupling between TCP flows, the UDP encapsulation is just one means we came up with.

> connections to take the same path?

+1 on being confused…  supporting different destination IP addresses is definitely out of scope of the draft as it’s currently written.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Thu, 24 Mar 2016 09:20:07 +0100

> On 24. mar. 2016, at 00.52, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Is this a real practical problem related to this draft?

> effects of the different paths that packets can take.

I’ll have to disagree. The CM is one congestion control instance, and E2E TCP-like congestion control is based on the assumption of a single path. Things go wrong if this assumption is broken, e.g. when packets from a single connection are sent in different directions inside the network. This is not only due to possible reordering, but also because e.g. a congestion event on path 1 would make the controller reduce its rate, even if path 2 still has 3 times more capacity available.

> it's *based* on the idea that packets can use different paths, and that

I agree with everything you say here, but only in my research world of non-Internet-compatible things. However, for the Internet, TCP’s end-to-end congestion control really doesn’t allow us to admit that packet can take different paths. (*)   I remember Van Jacobson presenting in his SIGCOMM 2001 keynote how we’ve wrongly applied circuit-style thinking to the Internet, and how this really isn’t how things should be. I sat there, trying to understand what he says, because I thought that *he* did this to us?!?!   (I don’t blame him for having made a quick fix that worked, this was a great job!  - but, to me, folks should have understood early on that this can only be a temporary fix, not a “clean” solution: if anything has made the Internet circuit-like then it’s TCP’s end-to-end congestion control, I’d say. Well, now we’re kind of stuck with this end-to-end thing that tries to guess the right rate over a path that can consist of an arbitrary composition of link layers… “IP over everything, and then let TCP try to do the right thing over it”. Nonsense, I say! But super hard to fix...)

> before jumping into this water.

I don’t get it. Why do you say “everything that IP provides will ultimately be needed by this layer”? I can’t see why the implications would be as bad as you say, at least not for the draft we’re discussing here. It’s only 4 bytes of UDP header that map a few TCP connections to one UDP port number pair using a well-known UDP port… the way you talk about this, these must be the world’s most dangerous 4 bytes  :)

Cheers,
Michael

(*) Researchy side note here: I’m all for fixing this, but this would require putting congestion controls inside the network, which you said you didn’t like earlier in this conversation if I understood you correctly. Well, definitely this is a research topic, not an engineering one, at this stage - but I’m convinced that this is the right direction to pursue. Indeed, THEN we can admit that packet networks can allow packets to take different paths. Moving such stuff to the Internet would require making PEPs first-class network citizens, and doing away with the notion that taking care of congestion control (not necessarily reliability! so these are not your standard TCP splitters) is a security breach. I have no idea if that can ever be made to happen, and lots of homework about control loop interactions and such is necessary before even thinking further on this I’d say.

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 Mar 2016 10:34:49 -0700

On 3/23/2016 5:51 PM, Tom Herbert wrote:
> Oh, I guess I'm missing something.

Sorry - it was I who jumped over the "between the same IP pair" above.
If that's assumed, then setting the flow label is sufficient.

> If the (destination) addresses are

It wouldn't - that's what I had misread. I was just noting that merely
setting flow labels the same isn't enough - the IP addresses have to match.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 Mar 2016 10:35:56 -0700

On 3/23/2016 11:43 PM, Michael Welzl wrote:
> +1 on being confused…  supporting different destination IP addresses
> is definitely out of scope of the draft as it’s currently written.

I was focused on the potential for either source or dest IP to be
different. If neither is the case, we're in agreement.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 Mar 2016 10:46:01 -0700

...
> Is this a real practical problem related to this draft?

It is a real practical problem anytime you tunnel.

> I’ll have to disagree. The CM is one congestion control instance, 
> and E2E TCP-like congestion control is based on the assumption of a
> single path. Things go wrong if this assumption is broken, e.g. when
> packets from a single connection are sent in different directions
> inside the network. This is not only due to possible reordering, but
> also because e.g. a congestion event on path 1 would make the
> controller reduce its rate, even if path 2 still has 3 times more
> capacity available.

TCP has always handled reordering.

TCP congestion control treats the network as a system - if the net as a
whole drops packets, it backs off. If the net as a whole does not, it
increases. That's consistent with TCP CC. You don't need to assume a
single path.

> I agree with everything you say here, but only in my research world 
> of non-Internet-compatible things. However, for the Internet, TCP’s 
> end-to-end congestion control really doesn’t allow us to admit that 
> packet can take different paths. (*) I remember Van Jacobson
> presenting in his SIGCOMM 2001 keynote how we’ve wrongly applied
> circuit-style thinking to the Internet, and how this really isn’t how
> things should be. I sat there, trying to understand what he says,
> because I thought that *he* did this to us?!?! (I don’t blame him for
> having made a quick fix that worked, this was a great job! - but, to
> me, folks should have understood early on that this can only be a
> temporary fix, not a “clean” solution: if anything has made the
> Internet circuit-like then it’s TCP’s end-to-end congestion control,
> I’d say. 

Van has gone back and forth on this issue. The next keynote I saw him
give in Switzerland basically did a complete 180 on his Sigcomm keynote.

TCP CC absolutely allows us to assume different paths - it reacts to the
*network* as a transfer mechanism, i.e., where the "path" is through the
blob we call a network. The only real problem is when that blob's
properties vary, but that's always a problem for every feedback system.

NOTE: threads along these lines have been popping up on V6OPS recently
too; it'd be useful to check there. I'm not the only one who thinks we
should stop trying to act like we can pin paths merely by picking IP
addresses, flow labels, or doing DPI - all we ought to assume is that
everything within a flow is "treated the same way", which does not mean
that every packet is treated IDENTICALLY.

...
> I don’t get it. Why do you say “everything that IP provides will
> ultimately be needed by this layer”? I can’t see why the implications
> would be as bad as you say, at least not for the draft we’re discussing
> here. It’s only 4 bytes of UDP header that map a few TCP connections to
> one UDP port number pair using a well-known UDP port… the way you talk
> about this, these must be the world’s most dangerous 4 bytes :)

Look at the tunnels draft in INTAREA. When you add a header, you create
a tunnel. The implications of doing so on the E2E and HBH signalling are
important and very difficult to get right. It's not just 4 bytes.

> (*) Researchy side note here: I’m all for fixing this, but this
> would require putting congestion controls inside the network,

No, it really doesn't. It just means we assume that the net is the
"path", not that we know or have control over a specific set of links.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 09:04:48 +0100

> On 24. mar. 2016, at 18.35, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Just to ack: yes, we are in agreement.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 09:12:54 +0100

> On 24. mar. 2016, at 18.46, Joe Touch <[email-masked]> wrote:
> 
> 
> 

Even when the method avoids that packets grow in size? I’m surprised but I’m not going to debate this further: you provided a useful pointer (the tunnels draft in INTAREA), I’ll have to take a look and try to understand the problem. Anyway I agree that a solution like the v6 flow labe that Tom Herbert suggested is nicer; I have nothing at all against that and now plan to incorporate it in the next version of the draft, for the v6 case.

As for the congestion control related bits of your email, I made a mistake: I started this off saying “let’s discuss this in ICCRG”, but then I follow up a discussion here on stackevo. This is purely about congestion control, so it really belongs there. My next email in this matter will go to you, cc iccrg.

Cheers,
Michael

---

From: Alexandre Petrescu <[email-masked]>
Date: Fri, 25 Mar 2016 16:04:45 +0100

Le 25/03/2016 09:12, Michael Welzl a écrit :
>>>> Having one uniform layer at which multiplexing occurs makes
>>>> things easy. Having two means you have to wonder what the
>>>> other is doing all the time and whether the two correlate.
>>>
> Even when the method avoids that packets grow in size?

The size of the tunnelled packets may not be as big as expected problem.
  For example, additional 40bytes on wireless are not that bad as
initially expected, simply because the total size is already around
1280bytes before tunneling.

On another hand, this problem of multiplexing at multiple levels makes
it that there is no 'traceroute' program that meaningfully reports
number of hops through tunnels.  This is a universal tool for network 
debugging, maybe as loved as ping is.

Overlay networks grow for a while and then get levelled down into
'native' multiplexing level if I can say so.  For example IPv6 tunnels
over an IPv4 world moved to IPv6 native (3ffe disappeared a while back
and 6to4 recently; each had a different way of relating two multiplexing
levels).

For TCP in UDP there would be a myriad ways of mixing the slow
intelligence of TCP with the quick dumbness of UDP.  After trying them
all would one come back to TCP and UDP side-by-side as before?  Why not?

Is TCP-in-UDP as reliable as TCP?  Or is it more subject to packet loss
and longer retries?

Is TCP-in-UDP useful for video streaming from constrained devices, as
useful as UDP is?

Does TCP-in-UDP require rendez-vous points?

Alex

> I’m surprised but I’m not going to debate this further: you provided
> a useful pointer (the tunnels draft in INTAREA), I’ll have to take a
> look and try to understand the problem. Anyway I agree that a
> solution like the v6 flow labe that Tom Herbert suggested is nicer; I
> have nothing at all against that and now plan to incorporate it in
> the next version of the draft, for the v6 case.
>
> As for the congestion control related bits of your email, I made a
> mistake: I started this off saying “let’s discuss this in ICCRG”,
> but then I follow up a discussion here on stackevo. This is purely
> about congestion control, so it really belongs there. My next email
> in this matter will go to you, cc iccrg.
>
> Cheers, Michael
>
> _______________________________________________ Stackevo-discuss
> mailing list [email-masked]

---

From: Michael Welzl <[email-masked]>
Date: Fri, 25 Mar 2016 17:57:01 +0100

Please discuss this draft on the ICCRG list. I’ll keep stackevo-discuss in cc just this once as a reminder to folks here that this has moved, but please remove it in your response. Let’s try to cleanly transition to ICCRG if we can.

About your email, my general answer is: you seem to be reading more into this than it is.

You mention reliability, applicability to video streaming, etc.: this is still “normal” TCP. In fact a design goal was to minimize the changes that we need to do to it, so it’s a relatively easy code change. The intention is only to be able to combine the congestion controllers of multiple flows, as has been suggested before (CM, Ensemble sharing, ..). The outcome is that multiple flows are closer to being like one flow, and there should be some benefits there in terms of new flows immediately being able to use a larger cwnd, priority-based cwnd share assignment etc.

Since you mention tunnels, issues with traceroute, overlays, etc.: this is an e2e encapsulation. Same TCP source and destination endpoints as before.

Does it require rendez-vous points: it requires a well-known UDP port. The idea is that a client checks using UDP if the server is TCP-in-UDP capable (listening on the UDP port), and if not, falls back to TCP. This is achieved by “happy eyeballing” (sending out packets of both types in one go).

Cheers,
Michael

> On 25. mar. 2016, at 16.04, Alexandre Petrescu <[email-masked]> wrote:
> 
> 
> 
> number of hops through tunnels.  This is a universal tool for network debugging, maybe as loved as ping is.
>

---

From: Tom Herbert <[email-masked]>
Date: Fri, 25 Mar 2016 10:05:27 -0700

> debugging, maybe as loved as ping is.
I believe https://tools.ietf.org/html/draft-nordmark-nvo3-transcending-traceroute-01
is the proposed solution for that.

---

From: Joe Touch <[email-masked]>
Date: Mon, 28 Mar 2016 14:26:51 -0700

> is the proposed solution for that.

I would not recommend that doc as an example of how to handle this issue
- it has more than a few issues:

- the reason that IP hopcounts of payloads should not be decremented in
a tunnel is because hopcount is a measure of time at routers, not time
in links. Because a tunnel is a link, the encapsulated packets
experience no forwarding themselves. This has nothing to do with whether
the source and dest IP are in the same IP subnet (end Sec 1).

- TTLs are decremented when forwarding; it makes no sense to decrement
the outer TTL upon encapsulation. That decrement happens later in the
path inside the tunnel. Otherwise, a packet whose hopcount is 0 might be
discarded at the ingress, even though it can legitimately reach its
destination over the tunnel link.

- The decrement of the inner packet might be done to correspond to the
cost of the link itself, as noted. In that case, the error goes back to
the inner packet's source host -- never to the tunnel ingress. ICMPs go
to the tunnel ingress if the encapsulation header TTL falls below zero
along the tunnel path only.

- RFC1122 does not ensure that there is enough information to relay
tunnel ICMPs back to the origin source. Packets could be fragmented to
traverse the tunnel (lacking an inner header altogether) or could have
too much encapsulation to allow that.

These issues are all discussed in intarea-tunnels, which should have
been cited there. FWIW.

Joe