From ipoverib-bounces@ietf.org Mon Oct 03 06:41:39 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMNlL-0002iP-FH; Mon, 03 Oct 2005 06:41:39 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMNlK-0002iK-M0 for ipoverib@megatron.ietf.org; Mon, 03 Oct 2005 06:41:38 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA19467 for ; Mon, 3 Oct 2005 06:41:35 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EMNtm-0002qT-Ta for ipoverib@ietf.org; Mon, 03 Oct 2005 06:50:23 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Mon, 3 Oct 2005 13:41:28 +0300 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E306A27A@mtlexch01.mtl.com> From: Ali Ayoub To: "'ipoverib@ietf.org'" Date: Mon, 3 Oct 2005 13:44:46 +0300 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.8 (/) X-Scan-Signature: b2809b6f39decc6de467dcf252f42af1 Subject: [Ipoverib] RE: IPoverIB Digest, Vol 16, Issue 27 X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0184727418==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0184727418== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5C807.7894F462" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5C807.7894F462 Content-Type: text/plain Compiled successfully, Bill, you can do that easily using mib2c (it's a standard tool in linux). Or use an online mib validator, like http://wwwsnmp.cs.utwente.nl/ietf/mibs/validate/ Ali. >Thanks Ali, > >The next MIB on the hit parade is your favorite, the interface MIB, can >you give that a spin through mib2c. > >Anyone have access to smic ? > >Bill >>> Ok - I have started reviewing the latest -TC MIB to push it into WG Last >>> call. >>> >>> However what I am missing is a MIB compiler to do syntax checking. If >>> anyone has a few MIB compilers ( I have gotten burned just using one ) >>> could you mstrip the textual convention ID and the Interface ID and run >>> the MIBs through your compilers ? >>> >>> If you could get back to me that you can do it, and when it can be done >>> by >>> (end of the week ???) that would be great. >>> >>> Bill >> >> Compiled successfully using mib2c 5.2.1.2 >> BTW, the object IbIpoibClientIdentifier (MAC addr for IPOIB interface) >> seems >> to me reversed! it supposed to be QPN+GID (QPN in the first 3 LSBs) but in >> the TC MIB document it's GID+QPN. >> ------------------------------ >> _______________________________________________ >> IPoverIB mailing list >> IPoverIB@ietf.org >> https://www1.ietf.org/mailman/listinfo/ipoverib >> ------_=_NextPart_001_01C5C807.7894F462 Content-Type: text/html Content-Transfer-Encoding: quoted-printable RE: IPoverIB Digest, Vol 16, Issue 27

Compiled successfully,
Bill, you can do that easily using mib2c (it's a = standard tool in linux).
Or use an online mib validator, like  http://wwwsnmp.cs.utwente.nl/ietf/mibs/validate/

Ali.

>Thanks Ali,
>
>The next MIB on the hit parade is your favorite, = the interface MIB, can
>you give that a spin through mib2c.
>
>Anyone have access to smic ?
>
>Bill
>>> Ok - I have started reviewing the = latest -TC MIB to push it into WG Last
>>> call.
>>>
>>> However what I am missing is a MIB = compiler to do syntax checking.  If
>>> anyone has a few MIB compilers ( I have = gotten burned just using one )
>>> could you mstrip the textual convention = ID and the Interface ID and run
>>> the MIBs through your compilers = ?
>>>
>>> If you could get back to me that you = can do it, and when it can be done
>>> by
>>> (end of the week ???) that would be = great.
>>>
>>> Bill
>>
>> Compiled successfully using mib2c = 5.2.1.2
>> BTW, the object IbIpoibClientIdentifier = (MAC addr for IPOIB interface)
>> seems
>> to me reversed! it supposed to be QPN+GID = (QPN in the first 3 LSBs) but in
>> the TC MIB document it's GID+QPN.
>> ------------------------------
>> = _______________________________________________
>> IPoverIB mailing list
>> IPoverIB@ietf.org
>> https://www1.ietf.org/mailman/listinfo/ipoverib
>>

------_=_NextPart_001_01C5C807.7894F462-- --===============0184727418== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0184727418==-- From ipoverib-bounces@ietf.org Mon Oct 03 18:06:32 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMYS7-0001ZQ-QV; Mon, 03 Oct 2005 18:06:31 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMYS5-0001ZI-TU for ipoverib@megatron.ietf.org; Mon, 03 Oct 2005 18:06:29 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA18424 for ; Mon, 3 Oct 2005 18:06:27 -0400 (EDT) Received: from nwkea-mail-2.sun.com ([192.18.42.14]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EMYab-0001xh-AD for ipoverib@ietf.org; Mon, 03 Oct 2005 18:15:20 -0400 Received: from jurassic.eng.sun.com ([129.146.58.37]) by nwkea-mail-2.sun.com (8.12.10/8.12.9) with ESMTP id j93M6PHT014202; Mon, 3 Oct 2005 15:06:25 -0700 (PDT) Received: from taipei (taipei.SFBay.Sun.COM [129.146.106.178]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.5) with SMTP id j93M6OYp430224; Mon, 3 Oct 2005 15:06:25 -0700 (PDT) Message-Id: <200510032206.j93M6OYp430224@jurassic.eng.sun.com> Date: Mon, 3 Oct 2005 15:05:23 -0700 (PDT) From: "H.K. Jerry Chu" To: ipoverib@ietf.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: OPNBvsOimozWOTge71pi5Q== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6 SunOS 5.10 sun4u sparc X-Spam-Score: 0.0 (/) X-Scan-Signature: 68c8cc8a64a9d0402e43b8eee9fc4199 Cc: Bill_Strahm@McAfee.com Subject: [Ipoverib] 64th IETF - Vancouver, BC, Canada X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "H.K. Jerry Chu" List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Hi All, I just noticed the deadline for scheduling a WG meeting in Vancouver is NOW! If you like to meet and have something to present or discuss, please let me know right away. Thanks, Jerry IPoIB co-chair _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Mon Oct 03 18:18:19 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMYXq-0003Lh-SU; Mon, 03 Oct 2005 18:12:26 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMYXo-0003Hj-Ex; Mon, 03 Oct 2005 18:12:24 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA19201; Mon, 3 Oct 2005 18:12:21 -0400 (EDT) Received: from e33.co.us.ibm.com ([32.97.110.151]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EMYgK-0002A6-Q9; Mon, 03 Oct 2005 18:21:15 -0400 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by e33.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j93MAatX027741; Mon, 3 Oct 2005 18:10:36 -0400 Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by d03relay04.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j93MCnQm549568; Mon, 3 Oct 2005 16:12:49 -0600 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j93MC7ZO027504; Mon, 3 Oct 2005 16:12:08 -0600 Received: from d03nm122.boulder.ibm.com (d03nm122.boulder.ibm.com [9.17.195.148]) by d03av04.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j93MC7Hr027500; Mon, 3 Oct 2005 16:12:07 -0600 In-Reply-To: <200510032206.j93M6OYp430224@jurassic.eng.sun.com> To: "H.K. Jerry Chu" Subject: Re: [Ipoverib] 64th IETF - Vancouver, BC, Canada MIME-Version: 1.0 X-Mailer: Lotus Notes Release 6.0.2CF1 June 9, 2003 Message-ID: From: Vivek Kashyap Date: Mon, 3 Oct 2005 15:35:54 -0700 X-MIMETrack: Serialize by Router on D03NM122/03/M/IBM(Release 6.53HF654 | July 22, 2005) at 10/03/2005 16:12:26, Serialize complete at 10/03/2005 16:12:26 X-Spam-Score: 0.5 (/) X-Scan-Signature: 5ebbf074524e58e662bc8209a6235027 Cc: ipoverib@ietf.org, ipoverib-bounces@ietf.org, Bill_Strahm@McAfee.com X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0464427413==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This is a multipart message in MIME format. --===============0464427413== Content-Type: multipart/alternative; boundary="=_alternative 007C44408825708F_=" This is a multipart message in MIME format. --=_alternative 007C44408825708F_= Content-Type: text/plain; charset="US-ASCII" Hi Jerry, I'd like 10 minutes on IPoIB-connected mode. I'm about to submit the updated draft on IPoIB-CM based on the discussions that occured over the last few weeks. Notably, clarify on the MTU, describe simultaneous connections, and connection teardown. thanks, Vivek -- Vivek Kashyap Linux Technology Center, IBM vivk@us.ibm.com kashyapv@us.ibm.com Ph: 503 578 3422 T/L: 775 3422 "H.K. Jerry Chu" Sent by: ipoverib-bounces@ietf.org 10/03/2005 03:05 PM Please respond to "H.K. Jerry Chu" To: ipoverib@ietf.org cc: Bill_Strahm@McAfee.com Subject: [Ipoverib] 64th IETF - Vancouver, BC, Canada Hi All, I just noticed the deadline for scheduling a WG meeting in Vancouver is NOW! If you like to meet and have something to present or discuss, please let me know right away. Thanks, Jerry IPoIB co-chair _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --=_alternative 007C44408825708F_= Content-Type: text/html; charset="US-ASCII"
Hi Jerry,

I'd like 10 minutes on IPoIB-connected mode. I'm about to submit the updated draft on IPoIB-CM based on the discussions that occured over the last few weeks. Notably, clarify on the MTU, describe simultaneous connections, and connection teardown.

thanks,
        Vivek
--
Vivek Kashyap
Linux Technology Center, IBM
vivk@us.ibm.com
kashyapv@us.ibm.com
Ph: 503 578 3422 T/L: 775 3422


"H.K. Jerry Chu" <Jerry.Chu@eng.sun.com>
Sent by: ipoverib-bounces@ietf.org

10/03/2005 03:05 PM
Please respond to "H.K. Jerry Chu"

       
        To:        ipoverib@ietf.org
        cc:        Bill_Strahm@McAfee.com
        Subject:        [Ipoverib] 64th IETF - Vancouver, BC, Canada



Hi All,

I just noticed the deadline for scheduling a WG meeting in
Vancouver is NOW! If you like to meet and have something to
present or discuss, please let me know right away.

Thanks,

Jerry

IPoIB co-chair


_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
--=_alternative 007C44408825708F_=-- --===============0464427413== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0464427413==-- From ipoverib-bounces@ietf.org Mon Oct 03 18:57:00 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMZEy-0004ZD-OE; Mon, 03 Oct 2005 18:57:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EMWrM-00011k-2n; Mon, 03 Oct 2005 16:24:29 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08382; Mon, 3 Oct 2005 16:24:25 -0400 (EDT) Received: from [132.151.6.50] (helo=newodin.ietf.org) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EMWzs-00069G-Ji; Mon, 03 Oct 2005 16:33:16 -0400 Received: from apache by newodin.ietf.org with local (Exim 4.43) id 1EMWrJ-00077w-4P; Mon, 03 Oct 2005 16:24:25 -0400 X-test-idtracker: no From: The IESG To: IETF-Announce Message-Id: Date: Mon, 03 Oct 2005 16:24:25 -0400 X-Spam-Score: 0.0 (/) X-Scan-Signature: ea4ac80f790299f943f0a53be7e1a21a X-Mailman-Approved-At: Mon, 03 Oct 2005 18:56:59 -0400 Cc: ipoib mailing list , ipoib chair , Internet Architecture Board , ipoib chair , RFC Editor Subject: [Ipoverib] Protocol Action: 'DHCP over InfiniBand' to Proposed Standard X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org The IESG has approved the following document: - 'DHCP over InfiniBand ' as a Proposed Standard This document is the product of the IP over InfiniBand Working Group. The IESG contact persons are Margaret Wasserman and Mark Townsley. A URL of this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ipoib-dhcp-over-infiniband-10.txt Technical Summary An InfiniBand network uses a link-layer addressing scheme that is 20-octets long. This is larger than the 16-octets reserved for the hardware address in DHCP/BOOTP message. The above inequality imposes restrictions on the use of the DHCP message fields when used over an IP over InfiniBand (IPoIB) network. This document describes the use of DHCP message fields when implementing DHCP over IPoIB. Working Group Summary This draft is a work item of the IPOIB WG. It was also reviewed by the DHC WG and modified based on the results of that review. Protocol Quality This document was reviewed for the IESG by Margaret Wasserman. _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Thu Oct 06 15:50:31 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ENbl9-0006jC-6G; Thu, 06 Oct 2005 15:50:31 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ENbki-0006VP-2o; Thu, 06 Oct 2005 15:50:04 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA26662; Thu, 6 Oct 2005 15:50:00 -0400 (EDT) Received: from [132.151.6.50] (helo=newodin.ietf.org) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ENbtp-0007tf-Om; Thu, 06 Oct 2005 15:59:30 -0400 Received: from mlee by newodin.ietf.org with local (Exim 4.43) id 1ENbkf-0002oq-LF; Thu, 06 Oct 2005 15:50:01 -0400 Content-Type: Multipart/Mixed; Boundary="NextPart" Mime-Version: 1.0 To: i-d-announce@ietf.org From: Internet-Drafts@ietf.org Message-Id: Date: Thu, 06 Oct 2005 15:50:01 -0400 X-Spam-Score: 0.4 (/) X-Scan-Signature: b7b9551d71acde901886cc48bfc088a6 Cc: ipoverib@ietf.org Subject: [Ipoverib] I-D ACTION:draft-ietf-ipoib-connected-mode-01.txt X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP over InfiniBand Working Group of the IETF. Title : IP over InfiniBand: Connected Mode Author(s) : V. Kashyap Filename : draft-ietf-ipoib-connected-mode-01.txt Pages : 13 Date : 2005-10-6 This document specifies a method for transmitting IPv4/IPv6 packets and address resolution over the connected modes of InfiniBand. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-ipoib-connected-mode-01.txt To remove yourself from the I-D Announcement list, send a message to i-d-announce-request@ietf.org with the word unsubscribe in the body of the message. You can also visit https://www1.ietf.org/mailman/listinfo/I-D-announce to change your subscription settings. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-ipoib-connected-mode-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-ipoib-connected-mode-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2005-10-6114823.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-ipoib-connected-mode-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-ipoib-connected-mode-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2005-10-6114823.I-D@ietf.org> --OtherAccess-- --NextPart Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --NextPart-- From ipoverib-bounces@ietf.org Tue Oct 18 14:12:26 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ERvwo-0007pP-1C; Tue, 18 Oct 2005 14:12:26 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ERvwk-0007pB-V3 for ipoverib@megatron.ietf.org; Tue, 18 Oct 2005 14:12:25 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA20054 for ; Tue, 18 Oct 2005 14:12:11 -0400 (EDT) Received: from brmea-mail-3.sun.com ([192.18.98.34]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ERw8F-00067b-RL for ipoverib@ietf.org; Tue, 18 Oct 2005 14:24:19 -0400 Received: from jurassic.eng.sun.com ([129.146.56.144]) by brmea-mail-3.sun.com (8.12.10/8.12.9) with ESMTP id j9IICFk8027212; Tue, 18 Oct 2005 12:12:16 -0600 (MDT) Received: from taipei (taipei.SFBay.Sun.COM [129.146.106.178]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.5) with SMTP id j9IICF5f611564; Tue, 18 Oct 2005 11:12:15 -0700 (PDT) Message-Id: <200510181812.j9IICF5f611564@jurassic.eng.sun.com> Date: Tue, 18 Oct 2005 11:11:04 -0700 (PDT) From: "H.K. Jerry Chu" To: kashyapv@us.ibm.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: HnUj89S4VZ3g+u7rABiPfg== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6 SunOS 5.10 sun4u sparc X-Spam-Score: 0.0 (/) X-Scan-Signature: b5d20af10c334b36874c0264b10f59f1 Cc: ipoverib@ietf.org Subject: [Ipoverib] Comments on the latest IPoIB-CM draft X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "H.K. Jerry Chu" List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Vivek, The following is a list of my comments on draft-ietf-ipoib-connected-mode-01.txt, many are editorial. 1) Line 27 "Internet- Drafts" => "Internet-Drafts". 2) Line 48 "This document specifies a method" => "This document specifies an optional method" Suggest to spell out the "optional" part up front. 3) Line 92 Remove the non-ASCII character. 4) Various places in the draft has paragraphs describing IPoIB-CM as if it were an independent transport mechansim for IPoIB that is separate from IPoIB-UD. If the draft simply states upfront that IPoIB-CM is an extension (or add-on) to IPoIB-UD, many paragraphs become redudant and can be removed. E.g. paragraphs beginning at line 161, section 2.1, most of sec 2.2. Line 313 to 319, 516-518, ...etc. 5) The first five lines of 2.2 may give an impression that one QP may be used to connect to all remote nodes. 6) Continue from 4) and 5) above, sections 2.2 and 2.3 may be combined as follows. Every IPoIB-CM interface MUST have two sets of QPs associated with it: 1) An unreliable datagram mode QP 2) one or more connected mode QPs [IPoIB_UD] describes how a node can obtain 1) and, through the address resolution procedure, a remote node's link-layer address. Once the latter is known, an IB connection must be setup between the nodes before any IP communication may occur over an IB connected transport. [continue from line 226.] 7) Line 291, why not simply say the QPN is the same as the one from [IPoIB-UD] to avoid any confusion? 8) Line 350, change "The IB connection is setup using the Service-ID as defined above." to "The IB connection is setup using the Service-ID as defined in 3.5 below." 9) Line 317 contains garbled characters (looks like apostrophe), so does line 331, 373, 375, 377, 386. 10) Line 377, 378 QP => QPN 11) Line 383 change "To ensure that two IB connections are not setup between the peers" => "To ensure that two IB connections are not setup between the peers due to REQ crossing" 12) [nit] Section 3.3 should spell out how to numerically compare 20 byte link-layer address. 13) Section 3.4, add a stronger statement "IB connections created though IPoIB-CM are considered part of an IPoIB-CM interface. As such, they SHOULD be torn down when an IPoIB-CM interface is torn down. 14) Section 3.5 the paragraph "The Reserved fields MUST be transmitted as zeroes. It is dependent on the CM to ignore or check for zeroes in the Reserved fields. This is because some implementations of CMs require all ServiceIDs to be explicitly specified and cannot listen to a range of values." sounds awkward. If we allow implementations to either check for zeros or ignore on the listening side, we might as well not to say nothing. 15) Section 4.0 Do we expect ARP/RARP to be sent through IB connections ever? If not it should say so in section 7.0 and remove ARP/RARP from the table to avoid any confusion. 16) Section 7.0 Need to clarify ARP/RARP over IB connections. E.g., ARP reply is unicast. Can it be sent back through an IB connection? (Suggest no.) _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Tue Oct 18 18:07:51 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ERzcd-0006HK-Dg; Tue, 18 Oct 2005 18:07:51 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ERzcZ-0006H6-G0 for ipoverib@megatron.ietf.org; Tue, 18 Oct 2005 18:07:49 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03147 for ; Tue, 18 Oct 2005 18:07:39 -0400 (EDT) Received: from nwkea-mail-2.sun.com ([192.18.42.14]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ERzo9-0006AA-JL for ipoverib@ietf.org; Tue, 18 Oct 2005 18:19:48 -0400 Received: from jurassic.eng.sun.com ([129.146.108.38]) by nwkea-mail-2.sun.com (8.12.10/8.12.9) with ESMTP id j9IM7hh0020626; Tue, 18 Oct 2005 15:07:43 -0700 (PDT) Received: from taipei (taipei.SFBay.Sun.COM [129.146.106.178]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.5) with SMTP id j9IM7g4C691192; Tue, 18 Oct 2005 15:07:43 -0700 (PDT) Message-Id: <200510182207.j9IM7g4C691192@jurassic.eng.sun.com> Date: Tue, 18 Oct 2005 15:06:31 -0700 (PDT) From: "H.K. Jerry Chu" Subject: Re: [Ipoverib] 64th IETF - Vancouver, BC, Canada To: ipoverib@ietf.org MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: n/DQfcwQNUIOZwjtiWMf0w== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6 SunOS 5.10 sun4u sparc X-Spam-Score: 0.0 (/) X-Scan-Signature: bdc523f9a54890b8a30dd6fd53d5d024 Cc: Bill_Strahm@McAfee.com X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "H.K. Jerry Chu" List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Hi All, We've got a one-hour slot tentative on Tues evening. But so far only Vivek has requested for a 10 minute slot. Does anyone else plan to be there? If not, we might as well cancel the meeting. If you do have something to talk about, or any remaining issue that is better to resolve face-to-face, I'd encourage you to come and please let me know asap as this is likely to be the last WG meeting. I'd like to get the remaining work completed and wrap up the WG asap. Thank you! Jerry Sr. Staff Engineer Solaris Core Networking Sun Microsystems, Inc. U.S.A (650) 786-5146 >To: "H.K. Jerry Chu" >Cc: Bill_Strahm@McAfee.com, ipoverib@ietf.org, ipoverib-bounces@ietf.org >Subject: Re: [Ipoverib] 64th IETF - Vancouver, BC, Canada >MIME-Version: 1.0 >From: Vivek Kashyap >Date: Mon, 3 Oct 2005 15:35:54 -0700 >X-MIMETrack: Serialize by Router on D03NM122/03/M/IBM(Release 6.53HF654 | July 22, 2005) at 10/03/2005 16:12:26, Serialize complete at 10/03/2005 16:12:26 > >Hi Jerry, > >I'd like 10 minutes on IPoIB-connected mode. I'm about to submit the >updated draft on IPoIB-CM based on the discussions that occured over the >last few weeks. Notably, clarify on the MTU, describe simultaneous >connections, and connection teardown. > >thanks, > Vivek >-- >Vivek Kashyap >Linux Technology Center, IBM >vivk@us.ibm.com >kashyapv@us.ibm.com >Ph: 503 578 3422 T/L: 775 3422 > > > > > >"H.K. Jerry Chu" >Sent by: ipoverib-bounces@ietf.org >10/03/2005 03:05 PM >Please respond to "H.K. Jerry Chu" > > To: ipoverib@ietf.org > cc: Bill_Strahm@McAfee.com > Subject: [Ipoverib] 64th IETF - Vancouver, BC, Canada > > >Hi All, > >I just noticed the deadline for scheduling a WG meeting in >Vancouver is NOW! If you like to meet and have something to >present or discuss, please let me know right away. > >Thanks, > >Jerry > >IPoIB co-chair > > >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 19 02:35:07 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES7XV-0002aB-S5; Wed, 19 Oct 2005 02:35:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES7XS-0002Vj-RC for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 02:35:05 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA24470 for ; Wed, 19 Oct 2005 02:34:51 -0400 (EDT) Received: from e33.co.us.ibm.com ([32.97.110.151]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ES7j5-0000Qr-Qq for ipoverib@ietf.org; Wed, 19 Oct 2005 02:47:05 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e33.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9J6YlfN027531 for ; Wed, 19 Oct 2005 02:34:47 -0400 Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9J6Ylb5527910 for ; Wed, 19 Oct 2005 00:34:47 -0600 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9J6YlJV023891 for ; Wed, 19 Oct 2005 00:34:47 -0600 Received: from sig-9-65-39-77.mts.ibm.com (sig-9-65-39-77.mts.ibm.com [9.65.39.77]) by d03av04.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9J6Yk2v023872; Wed, 19 Oct 2005 00:34:46 -0600 Date: Tue, 18 Oct 2005 23:24:12 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: "H.K. Jerry Chu" Subject: Re: [Ipoverib] Comments on the latest IPoIB-CM draft In-Reply-To: <200510181812.j9IICF5f611564@jurassic.eng.sun.com> Message-ID: References: <200510181812.j9IICF5f611564@jurassic.eng.sun.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: 3fbd9b434023f8abfcb1532abaec7a21 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org On Tue, 18 Oct 2005, H.K. Jerry Chu wrote: > Vivek, Jerry, Thanks for the detailed comments. I'll incorporate and resend the draft. Some responses below. > > The following is a list of my comments on > draft-ietf-ipoib-connected-mode-01.txt, many are editorial. > > 1) Line 27 > "Internet- Drafts" => > "Internet-Drafts". yep > > 2) Line 48 > "This document specifies a method" => > "This document specifies an optional method" > > Suggest to spell out the "optional" part up front. ok > > 3) Line 92 > Remove the non-ASCII character. Puzzling. Am using text editor on Linux (vi) and nroff. Will remove it. > > 4) Various places in the draft has paragraphs describing IPoIB-CM > as if it were an independent transport mechansim for IPoIB that > is separate from IPoIB-UD. If the draft simply states upfront that > IPoIB-CM is an extension (or add-on) to IPoIB-UD, many paragraphs > become redudant and can be removed. E.g. paragraphs beginning at > line 161, section 2.1, most of sec 2.2. Line 313 to 319, 516-518, > ...etc. The idea is to delineate the differences between the option -CM and the required -UD mode. However, I'll relook at the wording. > > 5) The first five lines of 2.2 may give an impression that > one QP may be used to connect to all remote nodes. ok..will make that clearer. > > 6) Continue from 4) and 5) above, sections 2.2 and 2.3 may be > combined as follows. > > Every IPoIB-CM interface MUST have two sets of QPs associated with it: > > 1) An unreliable datagram mode QP > 2) one or more connected mode QPs > > [IPoIB_UD] describes how a node can obtain 1) and, through the address > resolution procedure, a remote node's link-layer address. Once the > latter is known, an IB connection must be setup between the nodes > before any IP communication may occur over an IB connected transport. > > [continue from line 226.] agreed..though one reason for going into more detail is to make sure the whole process is understood w/o any ambiguity. > > 7) Line 291, why not simply say the QPN is the same as the one from > [IPoIB-UD] to avoid any confusion? ok.. > > 8) Line 350, change > "The IB connection is setup using the Service-ID as defined above." > to > "The IB connection is setup using the Service-ID as defined in 3.5 > below." ok > > 9) Line 317 contains garbled characters (looks like apostrophe), so > does line 331, 373, 375, 377, 386. must be character set problem...need to verify and fix. > > 10) Line 377, 378 > QP => QPN ok > > 11) Line 383 change > "To ensure that two IB connections are not setup between the peers" > => > "To ensure that two IB connections are not setup between the peers > due to REQ crossing" ok > > 12) [nit] Section 3.3 should spell out how to numerically compare 20 > byte link-layer address. ok > > 13) Section 3.4, add a stronger statement > "IB connections created though IPoIB-CM are considered part of > an IPoIB-CM interface. As such, they SHOULD be torn down when an > IPoIB-CM interface is torn down. > yes, good suggestion. > 14) Section 3.5 the paragraph > "The Reserved fields MUST be transmitted as zeroes. It is > dependent on the CM to ignore or check for zeroes in the > Reserved fields. This is because some implementations of CMs > require all ServiceIDs to be explicitly specified and cannot > listen to a range of values." > sounds awkward. If we allow implementations to either check for > zeros or ignore on the listening side, we might as well not to say > nothing. ok.. > > 15) Section 4.0 > Do we expect ARP/RARP to be sent through IB connections ever? > If not it should say so in section 7.0 and remove ARP/RARP from > the table to avoid any confusion. ok..will remove. > > 16) Section 7.0 > Need to clarify ARP/RARP over IB connections. E.g., ARP reply > is unicast. Can it be sent back through an IB connection? (Suggest > no.) will do. yes, agree on the 'no'. Vivek > > > _______________________________________________ > IPoverIB mailing list > IPoverIB@ietf.org > https://www1.ietf.org/mailman/listinfo/ipoverib > > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 19 02:37:27 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES7Zn-0003CW-7P; Wed, 19 Oct 2005 02:37:27 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES7Zk-00037Z-W1 for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 02:37:25 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA24608 for ; Wed, 19 Oct 2005 02:37:15 -0400 (EDT) Received: from e32.co.us.ibm.com ([32.97.110.150]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ES7lP-0000V9-CF for ipoverib@ietf.org; Wed, 19 Oct 2005 02:49:29 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e32.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9J6bDhr023240 for ; Wed, 19 Oct 2005 02:37:13 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9J6bDb5518010 for ; Wed, 19 Oct 2005 00:37:13 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9J6bDLt023009 for ; Wed, 19 Oct 2005 00:37:13 -0600 Received: from sig-9-65-39-77.mts.ibm.com (sig-9-65-39-77.mts.ibm.com [9.65.39.77]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9J6bCTs022983; Wed, 19 Oct 2005 00:37:12 -0600 Date: Tue, 18 Oct 2005 23:26:38 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: "H.K. Jerry Chu" Subject: Re: [Ipoverib] 64th IETF - Vancouver, BC, Canada In-Reply-To: <200510182207.j9IM7g4C691192@jurassic.eng.sun.com> Message-ID: References: <200510182207.j9IM7g4C691192@jurassic.eng.sun.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: 14582b0692e7f70ce7111d04db3781c8 Cc: ipoverib@ietf.org, Bill_Strahm@McAfee.com X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org On Tue, 18 Oct 2005, H.K. Jerry Chu wrote: > Hi All, > > We've got a one-hour slot tentative on Tues evening. But so far > only Vivek has requested for a 10 minute slot. Does anyone else > plan to be there? If not, we might as well cancel the meeting. As such I've presented ipoib-cm in earlier meetings. Therefore, unless there is a need for a face-to-face I can skip giving a presentation on it. Vivek > > If you do have something to talk about, or any remaining issue > that is better to resolve face-to-face, I'd encourage you to come > and please let me know asap as this is likely to be the last WG > meeting. I'd like to get the remaining work completed and wrap > up the WG asap. > > Thank you! > > Jerry > > Sr. Staff Engineer > Solaris Core Networking > Sun Microsystems, Inc. > U.S.A > (650) 786-5146 > >> To: "H.K. Jerry Chu" >> Cc: Bill_Strahm@McAfee.com, ipoverib@ietf.org, ipoverib-bounces@ietf.org >> Subject: Re: [Ipoverib] 64th IETF - Vancouver, BC, Canada >> MIME-Version: 1.0 >> From: Vivek Kashyap >> Date: Mon, 3 Oct 2005 15:35:54 -0700 >> X-MIMETrack: Serialize by Router on D03NM122/03/M/IBM(Release 6.53HF654 | July > 22, 2005) at 10/03/2005 16:12:26, Serialize complete at 10/03/2005 16:12:26 >> >> Hi Jerry, >> >> I'd like 10 minutes on IPoIB-connected mode. I'm about to submit the >> updated draft on IPoIB-CM based on the discussions that occured over the >> last few weeks. Notably, clarify on the MTU, describe simultaneous >> connections, and connection teardown. >> >> thanks, >> Vivek >> -- >> Vivek Kashyap >> Linux Technology Center, IBM >> vivk@us.ibm.com >> kashyapv@us.ibm.com >> Ph: 503 578 3422 T/L: 775 3422 >> >> >> >> >> >> "H.K. Jerry Chu" >> Sent by: ipoverib-bounces@ietf.org >> 10/03/2005 03:05 PM >> Please respond to "H.K. Jerry Chu" >> >> To: ipoverib@ietf.org >> cc: Bill_Strahm@McAfee.com >> Subject: [Ipoverib] 64th IETF - Vancouver, BC, Canada >> >> >> Hi All, >> >> I just noticed the deadline for scheduling a WG meeting in >> Vancouver is NOW! If you like to meet and have something to >> present or discuss, please let me know right away. >> >> Thanks, >> >> Jerry >> >> IPoIB co-chair >> >> >> _______________________________________________ >> IPoverIB mailing list >> IPoverIB@ietf.org >> https://www1.ietf.org/mailman/listinfo/ipoverib >> > > > _______________________________________________ > IPoverIB mailing list > IPoverIB@ietf.org > https://www1.ietf.org/mailman/listinfo/ipoverib > > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 19 05:01:20 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES9p2-0004gh-Do; Wed, 19 Oct 2005 05:01:20 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ES9oz-0004g4-8z for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 05:01:18 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA00843 for ; Wed, 19 Oct 2005 05:01:08 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESA0e-0004DN-4x for ipoverib@ietf.org; Wed, 19 Oct 2005 05:13:22 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZ3XL>; Wed, 19 Oct 2005 10:59:55 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4C@mtlexch01.mtl.com> From: Dror Goldenberg To: ipoverib@ietf.org Date: Wed, 19 Oct 2005 11:02:06 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.2 (/) X-Scan-Signature: 86f85b2f88b0d50615aed44a7f9e33c7 Subject: [Ipoverib] ipoib-cm nits/clarifications X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0448433478==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0448433478== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D48C.2BFA9BA2" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D48C.2BFA9BA2 Content-Type: text/plain Technical comments: -page 7, "The node receiving the IB connection request, however cannot determine the initiating node IP address". I don't think it needs to know the IP address. It needs to know the Link Layer Address. (The solution you propose solves the problem of knowing the peer link layer address.) - page 7, section 3.3 - Isn't it worth at least noting that simultaneous connections are handled differently than the IB active-active. This can be helpful to avoid confusion. - page 7, section 3.3 - need to be explicit about the numerical comparison of link layer address. It is unclear who forms the MSbits, it is the QPN or the GID. - page 7, section 3.4, worth noting the following issues: * since connection teardown is out of band in IB, some messages in transit can be dropped as a result of teardown. * simultaneous teardown can also happen (and is supported by the IB architecture) - page 10, section 6.0, I don't think that all CM messages should include the UD QPN and the receive MTU. You need both of them in REQ. You only need receive MTU in REP. And that's all. The rest are supposed to be persistent because of the Communication ID being used. This worth some discussion. I don't think that the current definition is broken, I just think that it has some redundancy. - page 10, section 6.0 - what is the final MTU of a connection ? is it the minimum between the two "receive MTU" , or is it hybrid - different MTU per half connection ? Some nits: - page 6, 4th paragraph, "the receiver", remove non ASCII char following it. - page 6, 8th paragraph, "the peer", same comment - page 6 "RTA" should be "RTU" - page 6, "The CM messages include...Service ID" - this is not true for *all* the CM messages. So maybe worth rewording. - page 7, paragraph before 3.3, after "node" you have non ASCII, 2 occurrences. - page 7, paragraph before 3.3, after "sender" you have non ASCII. - page 12, 12.0 remove non ASCII after "Author" My apologies if I'm repeating Jerry's comments... -Dror ------_=_NextPart_001_01C5D48C.2BFA9BA2 Content-Type: text/html Message
Technical comments:
-page 7, "The node receiving the IB connection request, however cannot determine
   the initiating node IP address". I don't think it needs to know the IP address. It
   needs to know the Link Layer Address.
   (The solution you propose solves the problem of knowing the peer link layer address.)
 
- page 7, section 3.3 - Isn't it worth at least noting that simultaneous connections are
    handled differently than the IB active-active. This can be helpful to avoid confusion.
 
- page 7, section 3.3 - need to be explicit about the numerical comparison of
   link layer address. It is unclear who forms the MSbits, it is the QPN or the GID.
 
- page 7, section 3.4, worth noting the following issues:
    * since connection teardown is out of band in IB, some messages in transit can
              be dropped as a result of teardown.
    * simultaneous teardown can also happen (and is supported by the IB architecture)
 
- page 10, section 6.0, I don't think that all CM messages should include the UD QPN
    and the receive MTU. You need both of them in REQ. You only need receive MTU
    in REP. And that's all. The rest are supposed to be persistent because of the
    Communication ID being used.
    This worth some discussion. I don't think that the current definition is broken, I just
    think that it has some redundancy.
 
- page 10, section 6.0 - what is the final MTU of a connection ? is it the minimum between
   the two "receive MTU" , or is it hybrid - different MTU per half connection ?
 
 
Some nits:
- page 6, 4th paragraph, "the receiver", remove non ASCII char following it.
- page 6, 8th paragraph, "the peer", same comment
- page 6 "RTA" should be "RTU"
- page 6, "The CM messages include...Service ID" - this is not true for
    *all* the CM messages. So maybe worth rewording.
- page 7, paragraph before 3.3, after "node" you have non ASCII, 2 occurrences.
- page 7, paragraph before 3.3, after "sender" you have non ASCII.
- page 12, 12.0 remove non ASCII after "Author"
 
My apologies if I'm repeating Jerry's comments...
 
-Dror
------_=_NextPart_001_01C5D48C.2BFA9BA2-- --===============0448433478== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0448433478==-- From ipoverib-bounces@ietf.org Wed Oct 19 06:08:24 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESArw-00076G-BI; Wed, 19 Oct 2005 06:08:24 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESAru-000769-Hv for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 06:08:22 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA04104 for ; Wed, 19 Oct 2005 06:08:13 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESB3b-0005yV-Ga for ipoverib@ietf.org; Wed, 19 Oct 2005 06:20:28 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZ30M>; Wed, 19 Oct 2005 12:07:00 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> From: Dror Goldenberg To: ipoverib@ietf.org Date: Wed, 19 Oct 2005 12:11:48 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.2 (/) X-Scan-Signature: 386e0819b1192672467565a524848168 Subject: [Ipoverib] ipoib-cm multiple connection model X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1697557823==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============1697557823== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D495.84040A00" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D495.84040A00 Content-Type: text/plain I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror ------_=_NextPart_001_01C5D495.84040A00 Content-Type: text/html Message
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
------_=_NextPart_001_01C5D495.84040A00-- --===============1697557823== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1697557823==-- From ipoverib-bounces@ietf.org Wed Oct 19 12:58:08 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESHGS-0001Fh-89; Wed, 19 Oct 2005 12:58:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESHGP-0001Eg-2d for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 12:58:06 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28514 for ; Wed, 19 Oct 2005 12:57:55 -0400 (EDT) Received: from brmea-mail-4.sun.com ([192.18.98.36]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESHS6-00017F-OK for ipoverib@ietf.org; Wed, 19 Oct 2005 13:10:15 -0400 Received: from jurassic.eng.sun.com ([129.146.17.55]) by brmea-mail-4.sun.com (8.12.10/8.12.9) with ESMTP id j9JGvweT002728; Wed, 19 Oct 2005 10:57:59 -0600 (MDT) Received: from taipei (taipei.SFBay.Sun.COM [129.146.106.178]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.5) with SMTP id j9JGvwwO298491; Wed, 19 Oct 2005 09:57:58 -0700 (PDT) Message-Id: <200510191657.j9JGvwwO298491@jurassic.eng.sun.com> Date: Wed, 19 Oct 2005 09:56:46 -0700 (PDT) From: "H.K. Jerry Chu" Subject: Re: [Ipoverib] ipoib-cm nits/clarifications To: ipoverib@ietf.org, gdror@mellanox.co.il MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: XePRFgbl4zWqF40SYA6dbA== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6 SunOS 5.10 sun4u sparc X-Spam-Score: 0.0 (/) X-Scan-Signature: bb8f917bb6b8da28fc948aeffb74aa17 Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "H.K. Jerry Chu" List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org >- page 10, section 6.0 - what is the final MTU of a connection ? is it the >minimum between > the two "receive MTU" , or is it hybrid - different MTU per half >connection ? Although there is really no reason to demand for a single MTU value per connection as long as a sender never sends a message large than the "Receive MTU" of the receiver, I'd expect most implementations would want to choose one value for both directions for simplicity. For that I'd suggest to take the min as the final MTU for both directions so that the side that advertises a 32KB MTU doesn't need to post 32KB receive buffers and waste memory if the other side will never send msgs larger than 16KB, for example. >I think that the current model for multiple connections establishment >between >two peers works well for 1 connection but is broken for more than one >connection. >Because of cases where CM packets are dropped or travel on different VLs, >they >might get reordered on the way and observed in a different order than were >originally >sent. This breaks the current model which assumes that the REQs are sent and >processed in order. Isn't this a problem at the IB layer already, or this problem is introduced by the 20byte comparison for the REQ crossing case? I wish the connection setup can be kept as simple as possible, e.g., active-active as you've suggested. But that requires a common service-ID, and the logical choice of using the P_Key takes away the flexbility of allowing multiple IP subnets/IPoIB-UD QPs to occupy the same partition... Jerry _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 19 13:17:10 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESHYs-0000je-4c; Wed, 19 Oct 2005 13:17:10 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESGyX-00029Q-Eo for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 12:39:39 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27448 for ; Wed, 19 Oct 2005 12:39:28 -0400 (EDT) Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESHAJ-0000Yi-16 for ipoverib@ietf.org; Wed, 19 Oct 2005 12:51:47 -0400 Received: from tsx2.cup.hp.com (tsx2.cup.hp.com [15.13.185.29]) by palrel10.hp.com (Postfix) with ESMTP id 4DC07AA4; Wed, 19 Oct 2005 09:39:33 -0700 (PDT) Received: from expvr082905.hp.com (vr724251.cup.hp.com [15.13.108.169]) by tsx2.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id JAA21914; Wed, 19 Oct 2005 09:39:33 -0700 (PDT) Message-Id: <6.2.3.4.0.20051019093745.02e59690@tsx2.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Wed, 19 Oct 2005 09:39:31 -0700 To: Dror Goldenberg , ipoverib@ietf.org From: Vandana Rao Subject: Re: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> Mime-Version: 1.0 X-Spam-Score: 0.0 (/) X-Scan-Signature: 10ba05e7e8a9aa6adb025f426bef3a30 X-Mailman-Approved-At: Wed, 19 Oct 2005 13:17:06 -0400 Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1375933318==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org --===============1375933318== Content-Type: multipart/alternative; boundary="=====================_89815618==.ALT" --=====================_89815618==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >I think that the current model for multiple connections establishment between >two peers works well for 1 connection but is broken for more than >one connection. >Because of cases where CM packets are dropped or travel on different >VLs, they >might get reordered on the way and observed in a different order >than were originally >sent. This breaks the current model which assumes that the REQs are sent and >processed in order. > >I think that we need to think of a model which is robust to CM >packets reordering >which can be one of: >- Adding a "channel ID" in the private data. The field goes 0,1,2 >and on. It helps > synchronizing both peers on which is the channel at question now. The rest > of the decision (per channel) of whether to accept or reject can > be the same > as defined in the draft. >- Not supporting multiple connections between the same peers. > >-Dror >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib --=====================_89815618==.ALT Content-Type: text/html; charset="us-ascii" Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
--=====================_89815618==.ALT-- --===============1375933318== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1375933318==-- From ipoverib-bounces@ietf.org Wed Oct 19 13:32:11 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESHnP-00065G-Rs; Wed, 19 Oct 2005 13:32:11 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESHnO-00064a-72 for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 13:32:10 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00203 for ; Wed, 19 Oct 2005 13:31:58 -0400 (EDT) Received: from e31.co.us.ibm.com ([32.97.110.149]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESHz8-00023E-57 for ipoverib@ietf.org; Wed, 19 Oct 2005 13:44:18 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e31.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9JHVrTd022590 for ; Wed, 19 Oct 2005 13:31:53 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9JHVrLk473044 for ; Wed, 19 Oct 2005 11:31:53 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9JHVqlc005414 for ; Wed, 19 Oct 2005 11:31:52 -0600 Received: from dyn9047018120.beaverton.ibm.com (dyn9047018120.beaverton.ibm.com [9.47.18.120]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9JHVqA0005365; Wed, 19 Oct 2005 11:31:52 -0600 Date: Wed, 19 Oct 2005 10:21:17 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: Dror Goldenberg Subject: Re: [Ipoverib] ipoib-cm nits/clarifications In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4C@mtlexch01.mtl.com> Message-ID: References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4C@mtlexch01.mtl.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: 14582b0692e7f70ce7111d04db3781c8 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org On Wed, 19 Oct 2005, Dror Goldenberg wrote: > Technical comments: > -page 7, "The node receiving the IB connection request, however cannot > determine > the initiating node IP address". I don't think it needs to know the IP > address. It > needs to know the Link Layer Address. > (The solution you propose solves the problem of knowing the peer link > layer address.) ok > > - page 7, section 3.3 - Isn't it worth at least noting that simultaneous > connections are > handled differently than the IB active-active. This can be helpful to > avoid confusion. > > - page 7, section 3.3 - need to be explicit about the numerical comparison > of > link layer address. It is unclear who forms the MSbits, it is the QPN or > the GID. yes, will add that. > > - page 7, section 3.4, worth noting the following issues: > * since connection teardown is out of band in IB, some messages in > transit can > be dropped as a result of teardown. > * simultaneous teardown can also happen (and is supported by the IB > architecture) I left the details out since it should be expected..however, we can add this information. > > - page 10, section 6.0, I don't think that all CM messages should include > the UD QPN > and the receive MTU. You need both of them in REQ. You only need receive > MTU > in REP. And that's all. The rest are supposed to be persistent because > of the > Communication ID being used. > This worth some discussion. I don't think that the current definition is > broken, I just > think that it has some redundancy. It is easier to go with one format so we can leave it as such. > > - page 10, section 6.0 - what is the final MTU of a connection ? is it the > minimum between > the two "receive MTU" , or is it hybrid - different MTU per half > connection ? The receive MTU gives you the peer's minimum value. An implementation can work this way. However, it might be useful to state it as the minimum of the two in the draft..since an implementation is likely to use the the same size buffers when sending or receiving. That is if A advt. recv of 32K it will very likely send packets which are only 32K even if the peer send a receive MTU of 64K. thoughts? > > > Some nits: > - page 6, 4th paragraph, "the receiver", remove non ASCII char following it. > - page 6, 8th paragraph, "the peer", same comment will do > - page 6 "RTA" should be "RTU" ok > - page 6, "The CM messages include...Service ID" - this is not true for > *all* the CM messages. So maybe worth rewording. ok.. > - page 7, paragraph before 3.3, after "node" you have non ASCII, 2 > occurrences. > - page 7, paragraph before 3.3, after "sender" you have non ASCII. > - page 12, 12.0 remove non ASCII after "Author" > yep..need to figure out why they got in... > My apologies if I'm repeating Jerry's comments... np. thanks for the comments. Vivek > > -Dror > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 19 15:33:32 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESJgp-0003dS-SW; Wed, 19 Oct 2005 15:33:31 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESJgo-0003dN-FJ for ipoverib@megatron.ietf.org; Wed, 19 Oct 2005 15:33:30 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08542 for ; Wed, 19 Oct 2005 15:33:19 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESJsa-00067Y-5I for ipoverib@ietf.org; Wed, 19 Oct 2005 15:45:40 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZRSB>; Wed, 19 Oct 2005 21:32:08 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA55@mtlexch01.mtl.com> From: Dror Goldenberg To: "'Vandana Rao'" , Dror Goldenberg , ipoverib@ietf.org Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Wed, 19 Oct 2005 21:37:11 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.6 (/) X-Scan-Signature: b1c41982e167b872076d0018e4e1dc3c Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1723269462==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============1723269462== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D4E4.802CA56E" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D4E4.802CA56E Content-Type: text/plain Hi Vandana, Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision. -Dror -----Original Message----- From: Vandana Rao [mailto:vandana.rao@hp.com] Sent: Wednesday, October 19, 2005 6:40 PM To: Dror Goldenberg; ipoverib@ietf.org Subject: Re: [Ipoverib] ipoib-cm multiple connection model Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib ------_=_NextPart_001_01C5D4E4.802CA56E Content-Type: text/html
Hi Vandana,
 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.
 
-Dror
-----Original Message-----
From: Vandana Rao [mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model

Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
------_=_NextPart_001_01C5D4E4.802CA56E-- --===============1723269462== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1723269462==-- From ipoverib-bounces@ietf.org Thu Oct 20 03:35:53 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxt-00086L-MG; Thu, 20 Oct 2005 03:35:53 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxr-000867-MD for ipoverib@megatron.ietf.org; Thu, 20 Oct 2005 03:35:51 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA21628 for ; Thu, 20 Oct 2005 03:35:41 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESV9k-0005c6-7x for ipoverib@ietf.org; Thu, 20 Oct 2005 03:48:09 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZT3A>; Thu, 20 Oct 2005 09:34:16 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA5A@mtlexch01.mtl.com> From: Dror Goldenberg To: Dror Goldenberg , "'Vandana Rao'" , "'ipoverib@ietf.org'" Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Thu, 20 Oct 2005 09:39:26 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C5D549.65D8E2EE" X-Spam-Score: 0.6 (/) X-Scan-Signature: 7a9d8f61f7718176ef97b85bbcc64331 Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C5D549.65D8E2EE Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D549.65D8E2EE" ------_=_NextPart_001_01C5D549.65D8E2EE Content-Type: text/plain After giving it some more thought, I think that there is still a problem. Please look at the attached pdf. In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline. On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly. I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine. -Dror -----Original Message----- From: Dror Goldenberg Sent: Wednesday, October 19, 2005 9:37 PM To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Vandana, Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision. -Dror -----Original Message----- From: Vandana Rao [mailto:vandana.rao@hp.com] Sent: Wednesday, October 19, 2005 6:40 PM To: Dror Goldenberg; ipoverib@ietf.org Subject: Re: [Ipoverib] ipoib-cm multiple connection model Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib ------_=_NextPart_001_01C5D549.65D8E2EE Content-Type: text/html Message
After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.
 
In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.
 
I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.
 
-Dror
 
-----Original Message-----
From: Dror Goldenberg
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
Subject: RE: [Ipoverib] ipoib-cm multiple connection model

Hi Vandana,
 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.
 
-Dror
-----Original Message-----
From: Vandana Rao [mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model

Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
------_=_NextPart_001_01C5D549.65D8E2EE-- ------_=_NextPart_000_01C5D549.65D8E2EE Content-Type: application/octet-stream; name="active active.pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="active active.pdf" Content-Transfer-Encoding: base64 JVBERi0xLjMKJcfsj6IKNSAwIG9iago8PC9MZW5ndGggNiAwIFIvRmlsdGVyIC9GbGF0ZURlY29k ZT4+CnN0cmVhbQp4nLVYWW8cNwxG73Za9DfocSbATiWKul6LBgWMPjTJFn2w85BuLgc+msRB/35J XcPxsdkmqRcGRJESPx4iufta6dmA0vxpi9358NPDoF68HV4PAHFOySunw5wMqvO+48uWOhswQLpO 2jXZjp8NL4c/76mL4TXp8VprtTHe0JqkQpWKBUBSv1wOD4Sgd4nWMQYBqMoaU4S7uNqgZ1NCqPA9 i0OVtkXaKP48/LUu3rwge+Ns+S/7Qa535+rnLZ91Ks104fb5oOnMhrE7D3NU0c8GU1Db82H8bNq+ Gu5vSQc5FY0hjXrWZAAB1jOi0z6y4j3cw9AEZfUKS2DvJAKp51TRHBOcjZ5BWwNu/HwiTvDO2/EL WmoIFm0cvxQiX/E6JnTWrGS+FjLf8L5DH4Mbv52KMh3G76aNmY0Di+PJhRD5Pmt1LvrxB7F98mYy s9Y2mfFHoUiKyLWUKXYEZxNd7nQmHm+Pitf9DE79Q3E9ov9X9H9KriY9MUUP6vYl5wGEpACc4cw9 H9DCQp4NjwoftZb8RjL/vTooqOQcz5dCuQMMbdvkbbmjkcgkEciEq4TvxI4FgxSMTEAlMscUQeMs sD5dFBBJIoCRScLCBmKoROycXcPZBV2/ZGXBjh40GBu7DWxSkG4BEx1x0RXyOd0cgfnNA+D5utTE geBTjUnZlsKLoRKuEzsWtEz6yjNSMBPdkuBZX/dAcB1e49lKRCawn0udJNBGCAoLsgcgogKrfTMp yMQBsKHjYQ8Aa11ywFqjyUOhAmxkPk0EJ0SASmAndizopGDOj1QJ5vjmKsZLsUwVD9vSQgsuLDkA 7IElByrOcg7Yby0HVhawBywFomNhk8rTCM0ky1Y0h3C94g9XwLw4rNglZXDWuKp3hgoe+x5gDqZU u4dUGUhfBBzvTzhTf4lhfDAFAK4PxlOt3v42bO8djycjVzTSE30cd7xG+vNmfDltcKZC48YnU5yN BufGi2nD9hobRjNV7sk00Wm+FuglBlAUrO3TjiHfWzDoCAUDFQNTcfi9QLzDeAOIRv8eIBsI3FR1 mu0KyzV//D4551n+E/shl1/Toms+MrpgyUuoEvTWejxusw1Adf9lU3/KMLWj8jy+5WVhq4lReuob f3czngnjnnWT3gjzl1PtqmQWTZdt8a5feUZbmFzA8Sn3oRuxfSJU7u5YS1gL2KvFLDUZTXdS3xOb ++wn0NISnG1EIwXPJxNmqhmwMqWzhZpV6Lsf295l2QNKzHcNzxqE09ffnTT9st95MfXFbeGSh666 z0/76vLm8cUDe5j2v2U824JABdrOGMsD2/YXfj0iYKE5g9l7MjK/9mf9je/NyJWmOzMSfbojI+Xl Uv2rfk7uynJ0tZi1oLrqxw42X2wK0ZqSDm61Reh+IiBdz0n23i0+WcO4MyXzlYek5C2uaUBP++qj U/KwHtDr7iHzJzX33Ipja9S0TeO6a42ahyqLWKaIwquzBw9VFk2bPejbGJG68FzoPCJ8P0UzBAYe DFxVQCRPJq6MG5WXKpHnVt3PpT7GEmjTeSsL8uxhA6tv8yeRDE3MHmySTZ989rA0AFJ3QhrsaNwS s8e1vn80kfspCPB/9v2DE8A6KxMAtef5L9XhFDWND0TmSbbxsBKhEzs+x+NgspWHUtD2KyhT+OsC 6fM1HPxVoqlvvFSJKBKAcOpOWv7aIoh1AqCOnAAGmkll/G4/QGjMJoV1Anx43OnCFncIN+K+dIM/ 5Ly3yT+vbGr4uW186uAf50B7HkJnpx4T0KcHfetF47QKrv2+k0nnm/ceLe/lQ92VsrtolujP5C5r ZVnfW3nPZCNqo5ERhbTtva8TyUuFwKrZtBj8NfWm1Lmr+3nK8kb0QXl778U5WA+GfwFB++kRZW5k c3RyZWFtCmVuZG9iago2IDAgb2JqCjE0MTEKZW5kb2JqCjI1IDAgb2JqCjw8L0xlbmd0aCAyNiAw IFIvRmlsdGVyIC9GbGF0ZURlY29kZT4+CnN0cmVhbQp4nK1YWW8cRRCWuFkQ8BPmcQZph66++xUS IRAvCYtASvJg1jnlI3ES5X/wi/mq77HH9hISK1J/2zV1d3V1vRrETHIQ/FcW+9PND/fd8PT15tVG Sj+HYAcj3BxID6f1F5t+Gk422slwGaolLJ+fbJ5t/vx+ONu8ghwrhBi2ZAlrULlM5ZMCYbhzvrnX EVoTsPbedQplWqJEXMmHrbZsinNZfcvkMlOrRE0D/93/OS8unsJePyv+F/3Qr/enw487/tYMYQbD 3ZONwDdb1t1YOfvB25l0cMPudDN+NO1ebO7uIANO1USQKGYBA6CwmLU2wnoWfMPuYdq4QYmFLo69 E6CkmEPW5sH47UQw3jg/fjNt1RyctHb8YookksL4HS+dNVaNH2MppFNa+fGTaStmKRRJM37Kax+0 UQSayuSzRCKtcuPn/KnR1jvTM/9y2tJMRio9PjyLJMqRCONXTejX3ZcPL6CsECoQlG269CRpbazV S5qPGsevJiMYuEe7X1Mg7CzN8A6h/hX/X8DzkOGDt3JYX3JaCGSulIY4kU83WiWY8/r3tK+FKPvS RVj3b5WBGJN0lpnKxIMMUll6GxIPQAnoogQAzUBmYOrOngkdQ5H3fGWSd6zLhNJLlieyAOkhT2of SdkAAJeBrzv7omclNJXJwoI9zjdp38SzSey2UNxCRrHiyaknmycbUkqy27IHpCBmF/I+INSXPjLL e95lYCrYM6FiaPMe9YQRFEuUtCyveEAhL0oQy57KwDPQ9btQIZSmjrCzgD2AM4NIKWGLSV0eQWLw rurDHpAsteUZzpUfVLDZYQVGcmmdAEhxBiDeCdkD1qkKAXRlUoDJhKxvy4EMU2gBRM0ByR5oOQDo OsKktOgBVQ9YbViiKiYtjoa0gfUxqnggXj+xIMbFYbUvDKRnoRflj6C4h++lnOGbWPzuoyrAQi/1 eHfSM64b78Z7k5OSawNZlO7db5vd9w/GhyNXM8jx1o97Xmv8szQ+m7Z6RpEx49HkZxx7Y8azacs5 Tyh8NOXdh9OEr5kt7J+dxNkYdsdVh8g36SC8TDqgGFDWw96oiDXaX1FEaHuLIlvp+I4VYVYLXa7z xwf2Qyy9VKJL/zO6EofW6gHXLdW7bRdtkCpAuyz+OaspUDrd+JqXaXuYWEuL++NlNeNxZ9zjatJF Nl/Bu+2rwipQk3ReFm8ryxP8pINxejzmO+hKbI86kftr1r1aTdk3zaxhInjC487rfjzY/tcgVB6X Z0d4OpGbFSgXptTtTswi9NWP5bfz9JuE694WfTrPQQkjLp+73vTzyvNsqou1cPUfvak+f15X51c/ bx64YVP9t4xnWzSK35bLoslH7Kd23I+607umyVlLpUtxRk3t4nx82QmRY0f6uBaEi+705rAaWWLN pFccGnk1nd5NhfLZNVLZydqGa/Z7BfqAt0PQ6X05a/m3nDBcJJsHrtsvWh+t2rJua7OwUR5QZ1cS suyerAWtxfSSS6rKV5mvJGnH+3iFbl+59Gr3aXeY3nPluHpKu6Rudp2CkbOIQ0u0tyt+6mLXu/lo Nak64hsP8S+dT++sBPT46tHqc6BLosb9nwkmOyvk8hQWls9u9uDLTsuSA7cfkceT1uiEgq8X5iEP ByVwoaOH8qnDUtzG4o2VHw4FxoYLgNtRbTKQFeyZUHR7uHYBbAa+sgAhHnksz2YBGUbxAI67PZGB r4C/U6LbS0qHHvjSNOLCZaUtVZNYm9I0AvFuemd80KZRac1tBZ4tCFTfNMbw7GrD9kffsG3juGSb yxpX/Q/YtaUsKOMJcfh4gvv1Sx1xkDN+Urie8PDi4cRf3XDigDzzqbV3+ZEr4gPE5jQoMMYQQNYH CIDuHieApt+z9XFSQElInx4nJeoZpvz08XFidALpVaXLd/FxUvei0rYHruSZFjqmK1WT+seJFoF3 E6MPmmdIB84zCkgoc+Dj5OZHwXt25X1ZX1aitZ6qp+i711Y1uwav9ZFdGW+1fdGwNFa1VY5p+SCm oOXn02yGR/Dc8UHzGtSqMDhTBpURGtvmNTWQ7xs//IQzjy64xu+6Y37L7du2V/sGWrlHbmt/rrna Ooe34vP3Smex4M/vA0vdk6rnvgzWQUMuHiqQJ5EnQtqbgZzKJ4ysgretKTMvR3bgJ0uBAYlgfNlF FgKWawhnCQfKSF2mRZJnAKHCgF1dJylkOaw63yBSanDWFMoYBEElFSr0DtBnNaQ2+Fa5MjPhoRYc X2FgWGpGHNlQvargQcuwsPLIK8CilUcJBavybVAs16lyzwrJapgCreuUVKSiCWWXAgRpVb6VFheH LmrENoC0K1CbNEbM0HAUjC73u9GCYWogDHsq7wEoBqXuZih7QsqABejSCGjDhKJ1KKxruUiA2JIq ngI8qKWrdrLq9VvBgVSh+Uixy8LCg7b5lxbuTt6nRWyK3By5ClNctV9EvYZZG99HDmeG1SiCFOlO SWQbpyYVSBxIrYogwVmubT/15DlqzXIZp7r1DFg+A2WC6QKHytcDQ/CGrcSWDbRFZzAFKyfLLk/4 yAtdoWVo8vzTMHBzmYwCtCGqqyc6TkYB8gi1O+uxq0NNgXrJS6jKxnGe1fs95Rm55W37P5o5F5s5 SvbHIt0NL1YnHv1N1N4KdXzxZpKu9eb3Nv8CsKZxX2VuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoK MTk5NAplbmRvYmoKNCAwIG9iago8PC9UeXBlL1BhZ2UvTWVkaWFCb3ggWzAgMCA1OTUuMjIgODQy XQovUm90YXRlIDkwL1BhcmVudCAzIDAgUgovUmVzb3VyY2VzPDwvUHJvY1NldFsvUERGIC9JbWFn ZUMgL0ltYWdlSSAvVGV4dF0KL0NvbG9yU3BhY2UgMjAgMCBSCi9FeHRHU3RhdGUgMjEgMCBSCi9Y T2JqZWN0IDIyIDAgUgovRm9udCAyMyAwIFIKPj4KL0NvbnRlbnRzIDUgMCBSCj4+CmVuZG9iagoy NCAwIG9iago8PC9UeXBlL1BhZ2UvTWVkaWFCb3ggWzAgMCA1OTUuMjIgODQyXQovUm90YXRlIDkw L1BhcmVudCAzIDAgUgovUmVzb3VyY2VzPDwvUHJvY1NldFsvUERGIC9JbWFnZUMgL0ltYWdlSSAv VGV4dF0KL0NvbG9yU3BhY2UgMjkgMCBSCi9FeHRHU3RhdGUgMzAgMCBSCi9YT2JqZWN0IDMxIDAg UgovRm9udCAzMiAwIFIKPj4KL0NvbnRlbnRzIDI1IDAgUgo+PgplbmRvYmoKMyAwIG9iago8PCAv VHlwZSAvUGFnZXMgL0tpZHMgWwo0IDAgUgoyNCAwIFIKXSAvQ291bnQgMgovUm90YXRlIDkwPj4K ZW5kb2JqCjEgMCBvYmoKPDwvVHlwZSAvQ2F0YWxvZyAvUGFnZXMgMyAwIFIKPj4KZW5kb2JqCjcg MCBvYmoKPDwvVHlwZS9FeHRHU3RhdGUKL09QTSAxPj5lbmRvYmoKOCAwIG9iagpbL0luZGV4ZWQK L0RldmljZUdyYXkKMjU1Cjw3RjdENzlERUZGRjNDRUNGODA3MkQwRkNENjkxQzhGNkRBNkNCM0NB QzJDQzlDRTZFQTk3QThFMkVGRkE3NThECjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAK MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA+XWVuZG9iagoxMCAwIG9iagpbL0luZGV4ZWQKL0Rldmlj ZVJHQgoyNTUKPDBDMDg2NkU1RTZFNkY1NzcxMkUwRTJFMkZGRkZGQTgyODFBREY5QkQ4REU5RUFF QUU2RTdFOEUzRTRFNERGRTAKRTBGRDhCMjJFQ0Y4RkZFQUVDRUNERERFREZGNUY1RjVGREZFRjdF QUVCRUFERERFRTBFOEU5RTk5MDUyNEFFQQpFQkVCRTJFM0UzODA3RkFDREVERkRGRTlFQUU5RUJF Q0VDRUNFREVERjY4NDI4MjExRTczRkZGRkZGRURFRUVFCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw CjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAK MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAow MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwPl1lbmRvYmoKMTIgMCBvYmoKWy9JbmRleGVk Ci9EZXZpY2VSR0IKMTUKPEZGRkZGRjIxMUU3MzgwODI4NTAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA+ XWVuZG9iagoyMCAwIG9iago8PC9SOAo4IDAgUi9SMTAKMTAgMCBSL1IxMgoxMiAwIFI+PgplbmRv YmoKMjEgMCBvYmoKPDwvUjcKNyAwIFI+PgplbmRvYmoKMjIgMCBvYmoKPDwvUjEzCjEzIDAgUi9S MTEKMTEgMCBSL1I5CjkgMCBSPj4KZW5kb2JqCjEzIDAgb2JqCjw8L1N1YnR5cGUvSW1hZ2UKL0Nv bG9yU3BhY2UgMTIgMCBSCi9XaWR0aCA2MzcKL0hlaWdodCA0ODgKL0JpdHNQZXJDb21wb25lbnQg NAovRmlsdGVyL0ZsYXRlRGVjb2RlL0xlbmd0aCA2NTYxPj5zdHJlYW0KeJztnWl25CoMhctZQXkH Ob3/Rb6TlxpAulcSo+0K+tNdNmj4kADjSnK7LVmyZMmSJUuWLFmy5DDZM5lldcus3meZ7S8LX5Ms fE2y8DXJwtckC1+TLHxNsvA1ycLXJAtfkyx8TbLwNcnC1yQLX5MsfE2y8DXJwtckC1+TLHwtsh2D b/9MfLMCya1OG7Xush+Cb/tQfJMCkVavWr0yDSYFIq1eNf1UHFPwqUH7GHxTAtFWr1m9Og0WvgIB +CYEAqxes3pBHBMCQfiumH4ojgn4kNWPwTc8EGj1itUL4xgeyMI3wOr1qhenwfBAPgUfjmN0IGTQ rle9JI7BgTCrV0s/lgaDA1n4hli9WvXSOIYG8in4eBxDA+FWr1W9Br6BgRhWr5V+RhwDA1n4hlm9 UvVaaTAwkE/BZ8UxLhBz0K5UvWYcwwKxrV4n/ew0GBbIwjfQ6nWq14ljUCCfgs+LY1AgntWrVK+L b0ggrtWrpJ8bx5BAFr7BVq9RvX4aDAnkU/D5cYwIJDBo16jeQBwDAjnGan+JpMGA9DvGan85Bl9q 9dJH9pxZGlZvq5+Cz0i+kfgS1XfxYzHp/3ub7S4b8Vzg6xxIpprjO3/6Zc7m+G7jAln4+lkV+AYm fXfZ0hHH+LYBgfj4RljtLym+u8Q3LJB80CC+hy89rQ6QNM8Ivnv/QPZs0OQPpKZWz51+D8+31HGF r3/65VYxvitUbwZI/zj0oEDEsFj4zl29ewBf/2lIDBrEd4nJT3g6CZ9QK/Fla8eZqzeb8+4a3yOQ ztWbz7QU3/mrNysXiu/eOf3koGF8va0OkF1k1xx80qr6TRqZ1fNW7yYDkfiegfSt3ii+s29d8p3D jeHrPA1tkg7Gd/7J7+meoKXxda1epVPhy/dRZ02/TQVC8fWsozi+c1dvPuX9eKnwqULrbdXAdz95 9e6KDcXXsXr1oCl8t/5WB8jTudfK4eLrkX651Qi+c1avWPl+Lil8/Sc/YdXAd/LJ7+WthIXwdasj MGga39bb6gABZCbgAwpdfGdMvy0P5H8XNT5Qa21SgK+j1f6CXHTxNaefWOXv72spPnma0Gp1gOy5 ix6+TtX70pJYNfCddvLbRCD/X9T4YLwNUoKvn9X+AtNqOD65W0kuZvi23Or50u/taxBflzqSW8zk IsZ31uoVWO7JxQyfzNK29IODQfH1stpf5NQXwNeheuUqdU99SfHJB+Kz4duhfya+DnWEx8LEd87q fTuVzOEIX9dApFUHX1Ll50q/Tbr3e5niUyE3WBVrg4fvjNVLStLG15x+xCrCpw4U6q0OkF0Eck8v 5/jUrrrVqlLF8Z108ttkIC4+sZ2oE2LVwXe+6mXOjcXHNCF8+kSh1uoA2aVz2fUcn95Wt1m9va26 +PrMGd1FBZJdR/hA1O1Wi/Cdp3rV1DcFn96f5+7k+NSRwnnw7TKQe3ZD4NP76jqrNfhOOfml7kTx tU9+ymoU38mqNx3NvDBcfA11xNVAfPpM4Vz45O7+RyA+sDPsYzWALzvRqrE6QHYVSH4H4mue/NKe hfjOtXXhjvn46uuIW8X49Ip1jurVzxkvvzA+fTpSEQh4uslvYXzZkVa51QGyq0BC+Bonv6xfKb4z VW/qikimAL7a9DOsYnxgxTpD9W46kNc9jA8cjxQHYg2Bha95yu0tIJDXPQsfLrwGq+X4zlC9FosI vq0qEMsqwQcOM45Pv00H8naK4MsCqaoj9HAjXcL4Gqfc3gICCeIjwddarcF3fPXu2qkJ+LI+0irB h04zyqwOEBCIvKnw5YHU1FE9vnNNfuhp833XxAePEOqtFuI7R/XmLmJMIXxFgdj9GT50nFFidYDY eRTDV169LfjyoT62ejcQSACfXXxBq3A/nDjl4DvD1sXxKIavuI7yKlRWGT60Yh1bvTvwSN128ZUm Qq6uEN+JJr8NBZLcZ/jaJj8PfhG+I6vXK8IgvsJE8KxSfKjoj0y/3QlkDD6vs4OvesrtLiiQu7oP 8DVNfqJxI77jqhdOfTX4ihLBRR/Fd/TWZffcGYJPtC3AJ/L26MkPBqIbAHw+g0KrBfjkrHlU+snR Y5D64sMHBMAvgG+rtjpA/AoM4yuoXnyqo1uE8R1VvdI9PZYcn4QQTwQfvIevbsrtLXLsAAEPH3uc d62yB9nUMYBPNj6yerviCydCgHspvmPST3oH3jhyfNWTn2xXga9qyu0uAVcK8EXrSDYDg1aM74jq VYnfhi+YCJGiN/DVr1i9RTkHPDHwVU5+EeouvjNMftI5NJAl+GKJgPHlvQrwHVa9atxa8YXeV4ag G/jI1HkUPnsOt/Cx0y3bqmpThU+/VrKtDhDlG0Vk4fNIYKuqU96oBN9BWxc9ajPwsSMx6BvER17H zK5egk+4UYQvkAi98On3SpbVAaJcg7lj4WPndL5V+0ohvmOqVxnths+oo2AXCx9bseZWr055mDpl +NxE6IavdMXqLdozGLqFr2Ly0/e74ZtbvWPweYkQBG7iYy9kZqYfGLEe+JwHj2i5B/AdO/lFp30T X/Hkx3g34ptfvdqxPvjs6tVacHsTH30jQ4yOkGggXfGF2wfwHTr5Rac+G1+4GDOr4FIrvtnVC/zq hM968AjTtvHR1+rY6gAJB1KJD9YRP0/ELW18YOxnVS8YLRK2jS8OpKx1Mb65Dx5xB3rii+eqjY+/ V0dWBwhwqxs+ngjxxhF8aOqZk35orMiUb+PjJ6wgkALWV8AXqrlafKCOCto6+PiLdW11gCCv+uFj 1dsbX3TK7S40EBYzxVdckN7hU9q2At+M6uVTXxd85mYk0P3m4jPerKum/QU51RMfXoWQhiZ8R01+ PBDWluKLQ4lfvFXhm1a9cJxsPGX44uPDtHr4SiaC3lIyh/fDF274drEG3/jqhT6xofPwwaSiK6N/ 9JQq8PDBDBidfnCUaObH8PnnwdYLNerjn8YXfaFGn+84PpjLc6oXukQPOT18uCjJy7DAyVN6w8MX nUm7SpnlGD68yUvCts7kO+MbW704xYfgS67Gd9KpkxxfeMXqLdgjatjFh8HIqzHIoruL74DJD3rk HHH6+JyyNM9iqdJSfBOqF49Qd3yCjJWiyKqLr3Ap6iZWsqD2Lj5S97lG3LsZH86CkdWLHXLg+Pgc NBbijviGb13I+PTHlxErnfoC+KylfFz1WvigVR+f+Y5J/580gW66+PAkNC79sD/GlBvFZ1YmUX9V fDFfkg4V+BJmJS8/0lsWPjIkRocOQkaHL7wBfH5uEfUd8JUkQgch7nTAx+i4DUbgG1W9xB2XTT2+ e9l736yvic9cscakHxsbw2QAn/2Wyb/PHfXxTZ38/Fke34rg4y8kWNch+EZWL/HGmm+b8d2d24ZK E1/5bqhV2MgYK0cEn/2aaSy+mZOfOfW14nNe+LC73fGNq17mzAR80fN23dNBzLpivU3CFFvpHsEX eM80DB9Lhv7Va099zfjMt5xUs+lqFb5RDx7Ml3H4NqOjabUdX//qtfGZnSrxvXqH31ZkN218zn6y d/rRUWnG57/mpO5cDV/hHN6Ib+P9TIVN+MZUL3XFHKwu+OIvK/J+Nr65k5+dB034vMmvyJ23wkp8 Q6rXmfpm47OttuEbkX7Ukwn4iq2G8E2d/JxA7G4OPi+QyfgGVC8fkQiYWny8nx3hSfEVL4Gd8JUe y8XwTZz8uCP2SMXw2Ue/0/F1Tz8+Hs4Tdh98pW/EFr68c6HVGD53u876FQv3w154g/hwIMZx6bXw GepCXJrxyVsOl1Z8nav3GHxGT0ddEB/P4b5npoYbjp0ifHdwEb9pm4OvV/VyN7xhCuJDap7XjsDX tXqNsRiI79kFdPWsBvFNmvwMb52Ftwxffv9pFFgfj69n9RpeeFaC+ICeN6Ij8HX8soY1EqPx4Xue tig+o3a8rnFx8RlG6vG9r+gg++IzZqUe6Wc44e6PoviUovSCMnJBfKX2077V+LADrtUoPqt43L5B scZhHL60g6zezvgqQwuK5YOb4VF8SlNqVEYyEV979c7Ht5sSsBrGZ21b3c4hMUehFd/2ijBHE8NH 8RTiGzn5NU19BF8e3v8qsifcGD6gBeqvxNenei0XyvERJEX4nk0VvpeVMD5z39oDnzkGpfgCTNyW dxef7uH5Z91rq962qS+Oj1YkxxfVXYuvR/WaHrj6owlShC/v4YuFz1p6j8YXjS+F4uJTC29MPRFz +nH6BqR66iuI7VdJtCSr8P12KAyhAz5TA7MdLtm3FOArXTkyKzhAM76W6qVmKb7yoFIaLr7yhVcI DLAm+oDY/NXNyoh+tTz+dXQ049sLzr5084H4qsP51fL418WXt68U8YBSFb4vwhbH15B3Ty2Pfx1N 1SsHUnVz1o5GfHb3l+Vmdnt88uuIj73HUwlSW70xfB3iEPisd6KveLuYdd7WtKXfHtDdDV+BbN3w 3e21YyS+p+pqzztK7fzh4GuqXof9426F51XexB0uwmd/T6IFn9W34Cl1Crm3FPmTTLmWsipHKL5t 3ysenrpWa8j1gHgrVr3rjLxYAkP4ZqLL3S/Ah9zsjW+Ddm0XD5OIb1k4LNxq2+g7i3u6ZNnuHZB2 uUQfApPIQPdqu8SVEL6muDtKET4BsDYLND7PrPDrVMLdRJOQ+rJNeSZIRcKsuXKcjN2vYIJkDpe9 iq3lIKRtY+E9JbtfAQTZEviKoi6onHq51bOKygM6Cz061MWV6uCDJsB2DXSYbLnLDN8DWV31JhoA PbTw9o5ypIBI0NSYtC1Lv4Q5UKuNnr5olbwct/b/74a1+JDW90OOfWBxbtl8fMlhepHqfGj0qCQL 76DgpsjOF8EXvwp8jy53+iWwF75RgU2ULCQQ6ysPw/JQ5e3Vh0U0VRx8L4ylGqk8Mn1YQJPlGHzD wpku284nvyToEnUevuttVSzpis9RtP++KvwoCX03LqrLxTcwkMOk1+QX+IrnJ4o/ZfXRMzSIA6VT 1vxRep0Cd78j9rlihx6rXlvH4AAOlg6x/2F6Dr+Qhr9Mz+YXqV6r/3DnTyBG/I34hrt+CmlLnzb4 nyAt+Hjy/RV6TQh43wmOn0Qa0m8lX0v6reT7kf74/lDyNSTRSr7/pTL9VvL9SiWHhe8hdVVI8U3y +jRSh6+u1wdKXR6xTn+tditnsYXvIVX41srxlIWvTRa+FlnZ1yRr6WiTRWLJkiVLlixZsmTJko+V P3cy21P2v3ew3U/+f6o82omLCvlFPkuYbPJPwCx8cdlepz/54dqxXl1E0sOzfeErEvEbJBa+ElFH tx+CTx5M9+8gprmFrzDyvD38FVltMRwokkbxdylLTcBf0NYWw4FyIL5P2PdJGl4kHfFd9Jdh5VKI o5yeQK5/TeOl8d3KfqNcMbv/Bf+a0A/Bl6VH/Ju8Fb+jSPT6EHy3Knx1Fj4RX5hKZfItfL+y8EEJ Yqmu3YXvRzrj2z4QnxVLsJnV89Ozb+Erl5QLr96GU5KFb+GjsvA1yR4C8xP9wgdk4WuSDJ/1RwIX PiQZGAtfbcifj889ftsXPiJRfPeFD0mOD1fvtvAxyclwfNUh/yV8OJx94WOy53+pg7dZ+JAE8G0L HxWBBk1+Cx+XIL7bwgdF4EPx7GfEd5If9N7TvxBF8d1vfsjkbgW+TbXXLeJ/GHHoexWJT/uxhfBR DeX43NTcEm1e4/ZvE1r9FBuMz3U0dfKO7wTxue/0stz0Gu+5EDvvu+Lqc16Dum8An275vGj5qYyC WzF87EtZ8v5d2UX1iwghQxzfY9oHup9KcmXIhbsdsnIFkgrh06NPcNx1a81PE3objOHbfjuwuVXj kw1fEw3Hpz2pxWd+B1Oxkq0D9N6Ngvh+/9LVRtKvC75Xxw0oKcGnI4KdnjcYGQEo9/5Ojd311dvv cdSd/eTsEx+f/HbVgtKDm4QCfK9FASORseL4VXu5yMDvWCeWsoY/RyU/P45nnkU14YMrIfgavYsv UQSLQaDC1aebK1bSoria/rDnT9HeI/jY2vHeY1l8tZPl+Lb0io8P/d4u6XimAKlE3/FP/fjBt//s 0bcwvrtqYOITjwjV+LasFR5NQCxnCtoCVFCjaPf8/2Pb3IKPhvy+rPwxBx3pEq2wNYCLfOlaJZ+X fvfs0msUf+a+AD42+e26wV0q0GOXXAnikz9eblrL2uLHFBCMnX75eL8+BOe+WnzyUi0+eQHtgW55 +mksxigyV3TfnOX9ge8G5Y0PV2/ydE7vY39UFZr4VK35+MDirvG5z48q7C3X8/Pcse1s2/xu24QP +FiGz2CS+Qu9BBch/FD15r2ch7ZEFxhDfB9wMMiU4TOacAIaHz7jc3aTKuLfj7cXRC2N+HTtKs8H 4bOrkhyRGiE8L247CoedxkJ8+UJgVeFZ8eHOPEPenTUsg16KD6W2hw/E0oQPFVXmL8Ynr7JHKBvf LoAEBOLb0W3rmdsKpRafM32ZawJZtr2F56Zr1xHI5wB8ehaYhi/z8L2RC4mNb4O3R+CTz35t+KDN m4tvL61dgi9dwKyHVxRhJb5NKhqLjzzMFdZutqzosck8COKTzYL41JJXgk/oYysHWTuSuAtrl+BL V01tWnfIO9fhu20ko27wakd82YN0UfLlmxqpPH/8M/BBKcUnZDA+mOi98N1fasHdOfhwkxC+naoP 4OMeYSfr8Zn0yrbNSkgGDcGXBlKUfDk+OfnlDhTiKztxQY7Nw4dXZF/y/OqIz0paFS72awQ+8kqi Dz4xOPnKEcUns78GH5+MuuHbcf/O+NBNhI9PGRX48vFQDl8E305uqkM1c7otx5fTm4KPFLUnAl+u XRiZhO95eFC/7zNmMnqLd7HEwiemvkn4Xo0viC+rXnpvIL7t3XYuvrrqtfDRkMfhS5seha8k/Ti+ XR19DceX65uLr656Jb5U/Wx8Qt1h+Aqq18CnbHB85oBF8Ukwg/HR3X0LPrHnwrfG4JOX5uKjT4mm cEaz8SkuDed9JICbgY9GbYpqnVcvtjAEn7oyFV/lgWknfJbBGD6N5Th88eq18REXR+DTrabi23f/ F2KQ2IgjPr7YXqkQn44009YNn7rcH59uOwofCKsLPvZQCLTW7Jy1I3wFvw4+vnMh+H4vlKdfL3yG vVPhwzceu99ifBtoS5no66Hn7DJ870Zj8akdbR46D0c5cyp8XbKvJIanztyLWPrZ+G7kBsLH7Z0T n9SZH/I04KPPEoDrDq491ag2E/H5z2byIg3REhMfP5MAl3Tru2xjHb1a+PC2PojPOI/NVMp4QumH 8EETuW5wCfgp20zEx9II9oZutOOTjYFL6S5HRSm7xfDpiXUEPsaA9CMCQVfiS/Uk9WAsdhCfcYjA rxpzKe4tu+oYI+kHG27ahFCNrqWKsk/QIXnRwqdWyXp81BVttBWf6o/wicP99JrqFsRn15GTkyCN UAiy6121iFTvhhu6+MCjgRbVzcKn13BxwXg8uSF8OI3YJTtIJiY+0lhopvT0g3EU3w71UlDkquqZ NJOzNC4nzi03gatXNsZnCTT9zDg0PiONs844OmvfA1JLdYTxeOmH9/VkO5Plg59+9nOxvujiu8fw gVlMXdLeQ8+c9KN5au4GZQcSLbqPJmhYvUju0mc4F5jpp7cUOpxsGC16jMfNOwXkfLy7EJVeex8q gEYSHLKhE1XRQ/lDwhCCaiV3x2msvQIxgFvkor4sIoEDxcBDOnLdyLup5BatS/BtewE+lH5GP3gR jruchkg/MhxSgXXTUIUT0MUHn7B8L50bu4FPPwDSj7xfHonh9HnwQU/K8ellgX3M+xn4QBcaVEd8 N7YtCLmC9ixv4fj0piRXSPpZ+FQfHlRPfH5j7ks2CqCToUmq+FVrbG2wd7xy0qhCqiKPvj2kjzmp Ytv9zX/Ys0kkaqVDqD9KxElUu0q0j12yZMmSJUuWLFmyZMmSJUuWLFmyZMmSJUuWLFmypE7+Kfn+ vfGFL6tO6fWfPrf883faS5kWfRPhaqjzos0X1ZdZlnH+Ew2JduhB0p9cBgZzj7+Fcuz0TbBWQeTc JXm/j75J3FQNKRvgQgU+Ze/tdOaXivufkarUDQOf2yfkpgrnxu6kznfElxs08clq5TpTRaKj0acr PkP7UfhECfXGx6FQN1XLwfiifmF831nja+FLQ6Mwvi2fYQ/sl9IBGodQEFdAl1p8gAsygJthJSY+ 8imCL/Mb/V8r4vhoH+nJF3VTms4MfLNPQubhg8iugQ83Q+YG4vuXNGYxjMBntNTKHwYSFUbuzcX3 rf0eju9mtsTNzHI1NELvyI0KfP+0338XX9rLxZduXj4fX5Yt6Q1Di5Ss46OF8vtgfA8RHWggjkbZ Wd8wtNj4ks3fJHzAE4YP5Ekq3zcq0/B9vTReDl/xxmUAvnf6LXxIi4fvtXqcHp+K5vvGZCK+r8fl 0+MrmPwwPtjUwfdNmz4/PFw5Eb7UZ2liLD4jaobvd7y/J+EDjVVLrRya+6Y36/HRY1zU9H3537+p +DID58JX8ND2bpbX08KHtBB8b35pR+7Owvf4lN44PT4VyzduB8wNxvdMv/Qzd+cYfDqWb9gOmRuM 75l+aUfuzlXxZUJv1OD70vikfGdqtD3QBW+GbTdVSx7nefC9nj3q8Zl90M0gvm+qogs+rbYG383R ufDZ+L5snXX4DPdPi++balERSQOhaAm+sj7UTR6Oob0BH4sGaMnufgkHvuRNFi0zaPbRN4mbXImh XYoM3GRv+vwtdGSfpIWQOTpeVh/Fj7kp2/FJ47ZkyZIlS5YsWbJkyZIy+Q/TAqdgCmVuZHN0cmVh bQplbmRvYmoKMTEgMCBvYmoKPDwvU3VidHlwZS9JbWFnZQovQ29sb3JTcGFjZSAxMCAwIFIKL1dp ZHRoIDEwMDAKL0hlaWdodCAxMTAKL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyL0ZsYXRlRGVj b2RlL0xlbmd0aCAxNzgyPj5zdHJlYW0KeJzt3el22zYQhmHHTeK2tmPXpZuUlOj7v8tooWUtFgBi G8zwfZpfPdkOgg8LMSDXawDrbnW/Mky6eYEm3JJzwDrj0zk5B+zHnJwD6/WKnAPW3UrHsDjpFgbE ddIpLE+6iQFpXWd80b4i50B3T84B45YQc3KOhetWS8j5DbBk6+52Cb4BS/Vz4/syfAUW7Pnh8SHJ 28PD89i+N2DRhhSPw+ObdISDSLcyoNnwpGE2J+dAiv5JOsFhpNsJUE1Jzo83G31mSTufslL/ddv7 G0FA/6Zj1T6OX2L8Wcu/Wd0FeM3jcSPPoHjoU4wQzRm0zObj+LfPH5H+C/RXsKghKevwFD6UhI4G ORZF0r19ubRszjd+OXjHgJRxYDsUeAaDgukvnfPZ6wX/AoCkN+fVRs7Dgp6U9PTJ3kjOX+Ondune vlRDr+RMbYuck3PEULQ5HzPkPCHlWXIeF3SpnDv26uRcF0Wb840fJ5ypDwx/rZx/icp5qYCHzOXO p3POnPMkrj2KNucb3uCFZTNk4k1da6eGNTHGQUmOO5iLegZ36HDkXIKeo/Od0+5eKiAJuQnP0j+T rPmbfRDumXrzke7oSyed3HmkWwtQaFAWc3IOzDY8qdqcj+QcmO9OW8zJOTDbQM4B4wZ9Ma+Wc54c wwhdhXCT2KOtnCIPs6af3vfzD78E+MYzhj4tem3P2rfii11yFLzUKIKJOuiPHaS8x/HhowKH6I3S VfA6KV3BHlJF5yqtuxwHpv8VMchUGC18Y0L8PRbS3og7RbfUPhyXtO9r3F82P7YCC9qzh3/mCiMm 8FL3WMi5ASpj7rivFnJZLS3n3piTc3LemEFXWfuB5P1z3/tkzN0/d+7d+94ddVLeAmXXVw6uX0Xd rNwrzejX424s567XTPDeKA16jWdqW4Gr87Q4X494+HR+iGj8c7jUmJPzpVN5dL6TY4ucZYYtEszZ OZ2X2rAl+WWgg07XXBtyQi5j0Llo34jqgu0KDYcc6a6KaHpnc+rbgVAqC+Em0m0HKKHx+sqBdOMB Suh68eMZ6cYDVNC8OR/JORBEayHcRPrxsxDpXgNt1L358VSeg+mUQ+qLE+v/D/Ifc88+7g6vY6l9 wCfd8ZdFa73ru5JFbfNLbKZCmyrfWIkeofIPJI+vEd9qkO75y6LuBa9nflx4mZzcZfn8gsuv1HLY HLV4FevrUscAz4IhcoUgHYEl6HXP5q57LOW/f64t58lzvfNeqi/o5FyQynfIHBPNeY7i+qiYC+Xc MaVvlu6x91JJemlqb6l9IOfkHG7Kj853HDHXsW6vmPOy91J5z0SrNNe7vktN6ozJeV5+E0J8muhs z+PDwux61na+IXfm2jeJE/M6LMzmJc7PvTHJ6LOgnZ293336s4LTODOdm9X37n3y/gSfRfhKoqU7 Od569c/at3yzRUCHTZmMpJTrFu39jZBiUPqC1zPSzQg0TfOl8yPSzQi0bDAxm5NzwEV9gcxEuh2B ptmYzsk5cJWBQrhJ2slS1MnUvihEVM5H97Werqf8WkSxcXS+U6NUpVThSsI5vbc+NbqwzTNEXjtb LztGIIqVzfmY4ftq1L2Gjx/OISC67pXkF0POm8m5pvvnrqjz3aXWDIamc/33WMg5OS/C0OZ8lP0u 8uJynrRuv5p06UAYZeGW2oeW5/NyMW8v58znjbFSCDdpOecFn8PFJV4m59NlIlJeleqvr1yakbo5 s2yWCBY9f/NnNiy5nvieFAvEzeSOiZycFzKYKYSbhHfprNzRyMlxZp2FL4U5SPf65VH+9ZVLtbvs rJ5b708CjvS2Nucj9e3AJUsn53vSLQo0iJwD1tk6Ot+RblKgMXYuox7J8ahLIem+hHbZ25yPp+dq MZSevpU6bWNoMsBizkVq2ApW2ESVzOS7tJ409EQPH9LBMMXg5nw8/y6y81bLrMstkfWwOarxvgR8 QL1UDV3CKOBaX5DzWoy8rv3CQuvb28t5/HeRyXlGg43XtV8Qm7mDpm9yHpDzXTU2sjBX7/rOuVQn 5xVzzvdSG2DyWfvW8aZ8v0d/OQjMenrOrwaenJPzmsxO57L78wzTuaqcu5LuPeQj5xVYjfkYF5Ni ScqVt7n588Yz01n+9V244zEbua7D6jO4rbNO6JtU6rve3/P8LnWl9MIMvwccBpsn53vSjQs0wtYL Xs9INy7QCMvTOTkHtkyv2sk5sGP5IdxIzoE909P5+GzS1xr/wRLpIBb2E+m+QbnNv+F3024A3Ky7 +1vL1gDW3e39/cow6fYFWtCZTjk5B9bbmNuezsk5sI25dBALk25hQFxnfjon54D96ZycY/G6FTkH bLvZxFw6heVJtzIg6mYtHcEqpJsZkLWI6ZycY9k66QTW8Rv7MjG9CmVuZHN0cmVhbQplbmRvYmoK OSAwIG9iago8PC9TdWJ0eXBlL0ltYWdlCi9Db2xvclNwYWNlIDggMCBSCi9XaWR0aCAxMDAwCi9I ZWlnaHQgMjcKL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyL0ZsYXRlRGVjb2RlL0xlbmd0aCAz MjA+PnN0cmVhbQp4nO3ca5aCMAxAYVuCPMqjjCLgDLr/XeosIgnncL8d5Mc90Wp7uVgJsZDyeq0A WLOqvG4ubSo773GBU7LqPPZDGvPgPS5wSladTz8i3dB6jwucklXnzbfzfPOeFjgnm8pDvEvKLYdw gAuTzOs4VTLOs/ewwEnZZB4fi6ysc8CJRechbKUsnLUDXiw6b55JfvnQDrix6Hx6iBSsc8CNQeZx y/LXsc8BNwadN4Ps1Y2/yABuLDqfZWedA47UK4+vVsqO39QAR+qZv+d9zAOZA46UM685awf8KXce 4vPbOV/OAVfKnf9fR81kDvhSrTyE+5hyReeAL83M69gUUg5kDjjT7DzU2yorTz8C3jQ7f21JFt6Q AdwpZl5P78R1VOAANDPvub8CHIJa5iH0nZQtT7wC/vQ6j1nGlm0OHMAHNYCeFwplbmRzdHJlYW0K ZW5kb2JqCjIzIDAgb2JqCjw8L1IxNwoxNyAwIFIvUjE5CjE5IDAgUi9SMTUKMTUgMCBSPj4KZW5k b2JqCjI5IDAgb2JqCjw8L1I4CjggMCBSL1IxMAoxMCAwIFIvUjEyCjEyIDAgUj4+CmVuZG9iagoz MCAwIG9iago8PC9SNwo3IDAgUj4+CmVuZG9iagozMSAwIG9iago8PC9SMTMKMTMgMCBSL1IxMQox MSAwIFIvUjkKOSAwIFI+PgplbmRvYmoKMzIgMCBvYmoKPDwvUjE3CjE3IDAgUi9SMTkKMTkgMCBS L1IyOAoyOCAwIFIvUjE1CjE1IDAgUj4+CmVuZG9iagozMyAwIG9iago8PC9MZW5ndGgxIDEyNDU2 L0ZpbHRlci9GbGF0ZURlY29kZS9MZW5ndGggMzQgMCBSPj5zdHJlYW0KeJztenl8lNXV8Dn3WWee yWyZmWxklgxJkBAyEJaAkTyQZCBEzADBJISQCYssaklYREDSVEUkiIlirVgUrIC2tR+TgDoJtqCl 1tq+JRWL2qKkfbGxrSlpX7SLknnPnQQEq+/XP77f9/t+36/PzbnbOXc72z03CSAAJEALCBCqmJ83 HuLf2Mcou3np7Q2NQ+3cEQC4cOkd6733Ns1OpY6zAGL3LY0rbp/7rtUKIBG9eHLFbZtuGaJ3Pgdg D65c3rCs+6c1mQABPs+kldRhXq5OADAuovbIlbevv3N4vQGaP3zbmqUNQ+3sizTnqNsb7mxUtgo0 p7GFOr1fabh9+TA9H+dpXLNu/VA7EOL4xrXLG5c92k1V434AzSR1Q0ocDkGKmAUpALG+yzB4W6yP 4waXxv7E+mJ/kl4GO3srdlE6AabYGQBGJVz1xb7kYzewG+L4t+BLviE6+ANV//BlNFAB44HP8FX4 G/RhCtwJW4BBEnwIxSSZvVAROwx/AYR/wG9j78JE+F3sx7AR3o+1EVUpNMElGpsG+6CX+r4PN8Fv iDKRpDQGpsFD8AQcgGegB96F34IBUuF6GrsDfga/g7+jFDtJY13EnTS4DmbDBngRjsEZOE8bbwUN 3NTug34YQLtQFuuEdKJZBPVwB+yBAyxHmAd22AkdcAR+QvP3IcOU2KLYytjp2NvgBD9MhgIog+XQ CF+ndBBegONEeYpWeId20wd/xhSciXW4HqOCXxgrtMRaoJZ29w14DLpoj7+Ev8IlNONozMFF2IiP YZRtBg+Mglw65ypYR7rbAtvplC/AqzTfXxFxBD6OUXyflbJ/CEbBIzwm7BG6RRTrxV3EL4kkW0xj 58I8uAVupRNvga9RehCehv8FEeiGH8Cf4RMUsRxvxxj7keAQkoSwcCH2eCwSe4ekkAAWyKYd5EAe TKJUADqdsRqW0nwrYTWddSPcBc005zZKX4dvxvn/XZqb8/YlOEk7fZ1O9hb8mnj2G5LDx7QeoxUl dGAycSQbJ2MZrb8UV+CD+DB+D99iBjrNTcLtwr3CceFV4RdCv5gkThELxQ8klG6QR8ttg32DF2Lj Y0dj3bEBOqcACkk7HXy01xwYC0FKZbCQuFsPK4hvd1DaTBp3L+1xO9wP7fAw7fIZks7rcBrepL2d hfdI6/5Cu/srxBBQRRvtbSiNoD2Ox3zaZyHehBvxEXwGu/BH+Ab+iVmZnWWzcWwCq2Dz2RK2lK1g uwUmWIQMknC+UCCExSyxWlwmbhcj4kt0ApCs0jRpnnRA+qGcK98Lv4eL8MG1JkJWsQTujlfrVb/Y hVNZM0wny3kK9uLX8T5cDL3Mi4+BTHr1CnyHTrJYmPtpxyUZ78dcnIc9uAsnszRWC82IghkThHuE l8UHYaaQANtwNTNjNysV3hIOskT8CRslOOCYsAC34s+ZXbpB+iH7EXEokyTyK3EljBbCUC5cEB4W CkgKy8RCksw4sgWNTYEg/oU069uk+T1iH/4e/0za5mLZxM2zeAAPwE0skXS1FytZNQvgPZReIYu2 wo/hUdKUu+E1gTyqfn3RtKlTCiZPyB8/LpA3NndMzujrRmVnZY70Z/i8Hnf6iLTUlOQkl9ORaLdZ LeYEk2Y0qIosiQJDGFPqD4a9kaxwRMzyz5qVy9v+BupouKojHPFSV/Bamog3HCfzXkupE+Utn6PU hyj1K5Ro9RZCYe4Yb6nfG/mPEr83igvnVlN9V4m/xhvpj9fnxOtiVryRQA2fj0Z4S5NXlngjGPaW RoJ3rGwtDZfQfB2asdhfvNyYOwY6jBpVNapFgv7GDgxOw3iFBUundjBQE2hXkdn+ktJImb+EbyEi ZJY2LIuE5laXlqT5fDW5YyJYvNS/JAL+GRFLTpwEiuPLROTiiBJfxruKHwd2ejvGnGh9IGqFJeEc 0zL/soZF1RGhoYavYcuJzPSXRGZuPp+cOyaKhyqrI4biKEJldRfMjrV0lLWUlNTw1ezF1duvJk8T WkuTV3l5s7V1uzeyf2711Vgfz2tqaNLcMeXzqn20a3/pA15+jHnV8RPQpJicR5vkffyYQwde7i/l PeHV3ojBP8O/snV1mISV2hqBeZt8namz9a5YL8wu9bZWVvt9kaI0f01DyYgOB7TO23SkTPeWXYvJ HdNhtQ1xusNsGa6YEq6uLL+Ci9fi5LxGu77MauQ78peRikS8S720k2p/hGUW8Gx5AbQuLSAy+mqQ OLqK+BdutU7lgpAyrX5v60dAiuDv//DanobhHjnT+hHwKleXKypH+Mv1SE5OZPRorilKMYmWdjYt 3p6YO+aOSLm/0eqNlBPLIFRNg2qm5hHLfT4u5Z1RHZZQI9Iyt3qo7YUlaZ2g5+XURFiYY05cxjgX cEzLZcyV4WE/qfNR4EGVM6JmXfmxWF2JpSunRtD1P6CXD+HJfEq9HaKU2Rqqzmpo3ZmWFW59oIZE EyRTbG0N+r3B1nBrQzTWssTvtfpbO8rLWxtLw5ePFI2d2JkW0R+oWYnE1Ej+EDciicXVQhqrGaqx NIFq5fP95XMXVhcMCy0iZtJP2TJ/6bJVpEItS1aTvOin4QGuaL5Wa2T2xz5OxzKtPf6XMYKJEXBY I1gY3zZGIDGCJPiyiJBUQMhcfk7lrsGbANRb6W7qU2Lxk1/9VfMetpvumEKopJuZkR/Mo9sUxAUU MwmA02NwmC2CCEEvgUDUi2jMevLiCFZ2I+gELQQC7Ke8l4CBl82GAEEjQQvBCYIeApl6KmhcC5tL eZjy/QQ9BAK1QtR3gvIBAkbzzoMQAe2IzaIZZ1ENKL/caiFoJ9hPIBPlLJphFs1/LeYEwQCBSuNm 0riZtK+ZNPdMOtFMws6ksWHKWwjaCfYPYyRaa+Y1Y8QrI3oIegkG4nQhyvkMjZ+bRaZRQVopSNgg YYOEDRImSBig3EvweQqZ5g7S3EGaOxjnyWcj2wkiBCeuzGD93CyhOOYy7f5h2qtnVOL0l2n57CLN P4P47qU8TMBb+wkiBAME8vRkwhUTrphwxYQrpjGXe3irN95jxQ7wEgSwQ9cE76bAJn1T4yaxsRtr oQVrdReDzS2bGVS1VDEDbMApAxvQkKBFVej0GHihf9MDFqvFawlYxCntlv2WiOWEpcfSaxmwKAYL ejAPi1Ccsg8P43E8hefwAsaQMJJHypOKJMJIh6Xj0inpnHRBikmEobgzTygSCCMcpojtlHCO4oWY oBhAs2peLaCJFsWj5ClFCi2o7dci2gmtR+vVBjRln3JYOa6cUs4pF5SYouhR5tWPILRZ27xtgTa9 LdQWbmtsa2lrbzOG2wba2FDvibaetl5qKt7XA6+feF3YKe6UjonHJDFNTJPKxXJJvF68Xvqu+F1J rPDs8zCLx+NhFe59bmZxe9zMYHFbPExdk4JFKXoKgxRrCluTjEXJejKDZGsy8S0ZUojA2e5kRU7d ycBpdbI1jnYHK3LoDka+wEFEDnAyNXKL7Inccix2kTTCha92vqV4oviqnvTWfMW1IJzsWRC2Jbsg GCQXYLep+jF8l2oGfK6zOYcIn+1sDlPxTGfze57pGn4LVrDHwYNPYaX4bQrrXBTeV3Z+w+Xqwj1D lSgu7mx20ZDazuZcKqo7m+/jI2+GZmkCjQxRsLUJamhkOQ0456WRZVipG36S7vq0eZTn77Uv8QXg b1iJ2S+Odr3fXOw53zzdiC9RZw9WGubhHHoinaJZ3u1c4fqPLqJ7vHOq66dRrDxy0e96nZevjHD9 JMonbXe6jtExuoYnfZGIizvHup4n5NGjqa6DDVFpQqfnQO2xOPpp6qVt7OPLadR+EivtzxHGBU/Q cgs6K117+cC3Pa6HiGTUk3QeF7QTig9uo7nXdE5w7fjBlU1up67DnTe4WmiTwkud97m2Ek65k+Z2 wSaslCd0vudqpK7M+vhMt/OZOj2rmqdb4zw+BBvi5UFYOKqcU+A+qKKJPVjTueEZz/epWsVMFHW7 sPRI1eE0Yr2nc8Nxz3QrpsNCekUfJ8wIGrKM3kUuTKPaWHp7uDD1SNV5P1GnPF/1ls/1j4VdfPpO 11+ropjw4nWu0xsCnl9sjvK9/LyqK+MCx3VviKL2gida9Z7n0MKopBx5xvVNIk/UTaNdD9Nm7ifE 6s1dpiX4km5zLaIZglpQCqoLDPS1U1Cmj1faf6+0/0hpr1FGqhmqV3WrI9RUNVl1qQ7VrlpVs2pS jaqqyqqoMpXc9HXHSRWnEMwieI1AxEiiUM7K58/A8siJpVC+xBv5eL4/isa5CyOSfwZG7OVQXjkj UpBTTn5kXmRyTnlECdVWdyA+SNHB/fHgkM7N29vSeFzYBYjXbduVxsvYtl01NbguGVw5//wl8wzL Q5u6if9uUHLK51O1PV5NTo88Wj6/OvKd9JrIeF6JpdeUR9bM9y6q7iL39G5pSRe+x4ua6i7Bh+co jKR+wUfRaXnkmTgZrMD3iIyeQ+/FycS/wApOBivEv3Ay4v4QXS0NJ7oGXhCdsgNq43S1yo44nTSB 03W8saK0pGPFijhN9jx4I07zRva8q2hIKWlsSUdtbZwq836sjG+sMvN+ooLySEF8ps2biWbD5jgN tsLm+EybsTW++ZmfkSwcJvnoCslHcZLwZyRVQyTs2csk7FkiwUY63X/Nr+4M+oKlO0tof0IfbzXE W53NK4KlK/30CvmfyRpq/xWybniDTj1MCV8g5qEPvxTzxd/yGXhk8a/3bOEvgLC/dDlBOLLzjpXJ FLl5vR17fj38NMgKL1m6kpcNyyO/9i8viezxl3g7Fm/5AvQWjl7sL+mALaWV1R1b9OUlnYv1xaX+ hpKaI3VNS3Zfs9aOK2stafqCyZr4ZEv4WnW7vwC9m6Pr+Fq7+Vq7+Vp1el18LSxdxc0tVN2hwoya 4kVD5RGmGcl6wmm+mhkua+O0uCld70tuTusWAZ8FjYJxE73uEgg4Knd67nSOEiGOMvOH3zAqufl6 X1o3PjuMslK3jUx5WAQUfZSujOIHpSsj+k56wtATTeYdvx/uyKAO4B1/HOqI4h/8JbB43eJ18e+f KuvXE2xYt2EDNRdTdjXk5MRr6wm/DtetX8dJqbGBFxvW8cr6K4ki0DEA2EHBIf/dzpwOSYxioBNk 5RgGCIn4y+cFAYyyFMVxLwgCm21QRF5FKFNvvjU55ybrxcI5lwpvsn5cOMd6qRCKCim/xLNxgXyb z5bps/nG4LbB7+Cowbcl+AQmivtpQQjF+sQV0mkYAYf0kXuMe1xsFiuVS42zzGIBmyhPNAqjWJac ZRRcTldKSppg7cbtYMMFutGxwSQUYQUyjOLOF9uhFwZo71Hs1I2pG0RDvRttB63kS1+EgzrWczKW r6cnNVeoIZXVq19V29R96mH1uHpKPadeUGPkrLtwB6RbP26q679U13+xvw6KLl2sO19H+flxAayj D+owSRb9GdlZNuvkSfnjk1xJylj0Z8g2qyt//CThrj9OjsGxPz787Ld3/+31+xtyB5J21G179sA9 4QdYWu3H//n8L3Apbnwz2rrk5eDGr34w+NHgH//wdeLCRqBwW+oGE27Tn7IKHtWjhYR6tV5rF/ap +7Qe4Zx6TjOBYFEtWpDpQoVaobUIbWrbMCqBh4MVQkQQXWqK9iY7K75pOGuURWYUJYPRGFRLDOXG Mu0e1irep7YY2oyt2lvsjPqOZvOyPDHAikSdVYiVrFacb1hkXMpuFZcabjOuY1vEdYa7jF3sqHDU cMR4kv1UTFZ1gZ6B+s2aLqoGQ5sgOgRBNAqMtWlGh6YZNZX0RGIimgyyoNAdxxQjqB4zrjG3mfeZ T5lF0aAamaCpimZSevR6uU0+JQslMnpklKMsRbfWC23xqFYsEdAjoMA707ymHqAXXIi/06zghUYQ vRCIt7vwYUiwflzXtLY/NeXS2qY6DqnJ/dZCa6HNPmWKbcqUurWkkYVF1Eyawruo2C6Nzdm+9eT2 scm8gIKCAi7jprVAQBqbhj6b3yb4BdtGvPVXb+Ntv7nwh36p+9Nk4YN/BMW7P2nmwDV4K70oJ5Ps DPBzvZnLa79wWI0Ix9UBIaaqDiFN9Qp5akAoUrncuFi55C4TnRBOqVyGvcIF1cUlzxQmKMTOVari UFVFJWaKIkMmKxTeE1YVhR5Jl8PEtsOymCc3yvvk4/I5OSbLxD2b7gDV0INDPBKu5pCRcyhukvGC mMJZYi2MZ8SSYX4gVccFgDiYb8tHW77Nj7atH+KHv704OIKOnyuc/kdQeOnTEnIKW+jkR+jkTvDB Pt37qIAGS7K1TFuo7bHuSfmdVTEYrKguUzHBoJmtZIu1+hgtgZQkIdWABtfCPAeCAx0O8C1EZgbV fMybgAkJ/rRmfixGx0nVLeBzrhg+hpe4zU+SwU00fpB++5S8pn46R1G8Vtc/pW67eWyOtNXKzwFj 6SB1uLhucR0dxjfejU6HrLgxKdEnjGU5mIL546fhxAlZ2Vk5uAVjJ5+97eBjN91S/fDguUh9yY2V 448eXFhQEMh4+gdSd8UP73/ujbSCbc8N/haLvlflu7RXuCm9uiy4wKrRrshtynlx+/2p/rJVsMrc fkMyt992mdtvjxy3X5Atmi7oMrfeFrntMuI6+RmNuSFdTJSshifhCXG31G54D94VT0s9BpMBjKIo SQbNKJlmQ7lYKpUYVonLpVbYKW6T7jV8Q7K9A2cMv4ffGcTF4lzpK2KDJJ4ST0g/M540ifXGkGmN MWwSF5ieNf7NJCQbFhrfNn1gEpMgWUiSBVVQZYPWaCarjNeo4/s4ht5WcqwXhFivfqfmNZDyrVIN pI8GugGkTUOWLmom06Yh+weusFyYCIKsCWikTTNRUkRVRXq3qRXmenPY3G4WFbJ9SRZNGjABdTlE etwoi7KsaoKAVlVX2Uh1gtqonqSgOco8usWh6RobqU3QGrWTmqhRX4cX88jM65r665pI6GTNeXXD Vj5s65BMClH4ZZZ+bTFk9/QIPT+kJ0PfkBfwYT55gTTMR5+GOHLwN38/ehZ9p7suDp4dHBz4E1mD TbjwKQUB5BBmfRIliyiMfSDcLk4j0xuLI/SgBSSXlGRMafO1ZbT520Y+kLlt1BlJGylPkL+T/L7r /aSPXB8lKU65zF5jF84oaPFV+Np853xive+CL+YTPL46H/NFaaoiPTWUyljqjNT7Up9MPZwqpaa6 TQnG0Oje0QOjheBotI3G0VU2Db9WBGih51seCAYLoOoBpB+6Fqfrk+vdF9wxt+B2o7tKVCwBHJta e/UVKKs/THgyYAqZWkztpv2miKnH1GtSTVE2WU903E2znMKd3uxANstuaCS77cI/Qx45lf66i/1N 3BgvklE20Z25tu7i2qY4N88X9ZMjIbcbN861dU3kYpvoEm3C7GlkfGSQgsOV5M820xWaNXHCNJw8 afKE+H2KskLJ6XDFTRQzf5F1+MDDj89gLvd59/Xbb/zqt2bcee/+opTiG6bNRPeYult9vrLJ48KZ 7Ldjn9o1b1f3YLR1+43rg8HvPrjwvpAn01s92V8weMqePCIjY9IN84sXbQKS1iSS1jKSlgvc9KCb afFiq/lR9qgsrGKb2A52n7zNKG2U7zBuNj8miyvkW4yrzUJLUoubJYHb6tbdje4Wd49bDrnDVO11 D7glK7E2iqm612axV9jb7MIaO3rsaLc7U/kvdRh+7QRghMuDufSykDPsfNEp2J3orBK1NamYnoqp VSbFbc+DImC74F0gr4D0OHoqL6koiSU1BGwhW9jWaGuxtdsitgGbCjbdxmwkoeftDWtUVLlMPFwm l5riciHu2/KBpFDH4aO6pvPcVvLW8iuAZ6T6TfHApgmTuBA+i2tsis9NIkrk0lDkJLb2/bGv7Hx5 1V1bVz97bPVmvHSc3T5nzXhhWWnZ+HzEeXn7v/m1R9GMxv3bW58c/Jm3uRWP3rV1+ow7aP93UlD5 EHlHF2zQ81+T35ZZtjxZDjLBSmpo1DSH00U4MGm1BqsR84wVxnpjm/Gw8ZTxnPGCUTVGmVlPcoKj 1hVwoscZcDIvZboz5BSdXZgISfG7/7Orjd8H1vjV1h8P29DGlc2V5HLSjT4hO4vUi9ewfvK2m1Y+ 4Hb9LsVfWbO0T+q+9H6o4ivLZ3deeoiN++G4GW0nL31I2yZN2cT/h4CqPDZe2MEjklH6BFmSFdG4 xhAxMEFmCkgU6tRiQNTFkNgitouSVwyIYbFRFHmF94oi3656ebvcXdm5v+Lu6uOfjwsk0nUrEGzq 6+sTX0Bx8NNPZotZn/yKeLiCIsSVpKuZ0KVPH6lkJkxQJiaUKJXKeeV8wkXlYoJRVMQE5qzSNE+V pCDIVpstJTV1ZKYxLzuU3Z4dyRYt1lp7iDSHK0ySrqUmp9SmhVLbU1kqb+dl+kfWZuITAJAZyNQz Q5ntmVKAinBmI1VPZMre4f5wZk+mnNmFhZDFOb620NpPcb71Yl3h5bxuKLSIRxdF/YSP231ivnNI EMO2z6VxWRhxsZh534o3p9xfUbkle+q28pqvFfcVzJizuC8x5cb8m7P6xKyHFlRWLlhQueDJA5dq WP2+VbvPDDIW/N74GSX37L306VAkLTwUt+m7dT3TMdHBUhybHDscexzfdnQ55L870GJHk1GuspsT ZLtisWgmrM0DtALmQQXUQxschlNwDi6AClzxXFotd4nMYwqYmJcynVqi6TO96+cs6OdWBUX06unn 3i4edzTV+ehc3H7i57INnZ77vIf6UmfmTl2c1XdmSlvV8p0Tmfvp26aW3/vioEfM2rtw4aqDe0nn isk7vUonSYAUWKVPPqS+oLIfS29JTEMDXZhC0mbRtlkTFU3rSUNHwyylWgkrjUq7ElF6lAFFBSqY EmVWPcHc0Agt0A7CmngE9WdIpZ2vjfsI8tdFJL7+IUcAdYkT7GT+dqeDyf4MSOT+d9LEuJSK73pt 6xsDW3+66ZUYfNhcU3VXc3XVVpaxF6Fl8MWzBwb/tg1HofDUwUPfevLQIVpnDRnO07R/K2zR3XRJ SiwkhaVGSThnJxl47MxoIDWll4BsimKNPkZRjAYwoJpgOGdgBgN5TxJKLaLwRIi1M9bL0MsCTGch JjLOfdsQ9/kTk3xcHr9wuKfjtmTjf9mmWJ4uGt9lvicpQwqGN/edmVw1pWLWrCn5gTKvmPWN1aUT Pxo7s/vvtOdRxPObac/Z+Cf9dZn/n0/CduN99m0jdozsGfFm+mn3G57T3tO+hNnGOdoc05yE4vSg u9gzw1vqM2gJUvZEd1lCML2Eukqpa2b2/dq96fe4Wzz3eO/xva3x8T3x8a5MyxRtorfUvcC91r3W c9D9vPsn2juaNsI4QhthGpHgSE90Wz2J3kTfIuMibZFpUUJl+nx3yDPfO99nf9y4V9tr2pvwSPpu d7tnt3e376zxrHbWdDbhswXsT7hwRwpeb0GjId3jiTKL/j2D5jAYtF8bUNN2aMxkcBj8htWGXYZD hqOGs4azmmG0ocxQYxAMmiddRAc5XAHQjl4U7sajeBKFV/ANusFQcDqF+iRMqrKF6G1WlanZFMt1 qKUbPGKGOcF5l5PlOYuca5xfJe98ysFwHTIvBjCMjShiAzgcGfIZwEfhELzA33LmUXJDRuopyMBA RjijMaM9I5IhZXDLM60x95rZO2asMKOZa+0o68dr+bXWRPp6Fze45LqmodJKnRz4845rdH8dtSjo oB/SETt3s4Spa2pay58HZnoeWAuHX3tr6RpsakqcPKTo2VnZI7MpEpnE3ZQSd04OeuPzu8NBBpE1 qs+9d/48e2LB4Pk54ZeOfLdz5Dvp2xfM8vmee7m05HT3t89gXs4zer7f4bCUlyx45JHObY+MaxmX 7U9KDkyaM6fl8deeIS1LiX3AUqUnIBl26WNXme80s9HmqebZ5oVmMdkBSYLTAS6bPRFdNpaISQIF 04rLkaRsNmlJUXxezyEfltiWyBKj4JANxp3YBvvIX9yXJNS6bI7EV8HmtQUoLgjZJBs3lBQylIvk ii8VflpXeHE8ueZLTYVWCs2IK3TZ5FvoG7onnX4bnTZ//OQkflbig80/MX/iZBtTnlDMOd5CR2h5 9Wq7efVq8sS9g5U7XaPSfjW6smJqJ57qPX1gcEf8767X/x9MP/jnhHtZynB68nISvvXPSdw0lKTc eHpQOjWU5KXya8qoeLpbfZ5s4N/p3+nf6d/p/2oC/qoY+o8VB78F6X2RSiBTZdw8ISswcVbVgpK5 N4dunFr+pf/Y+//pJ/L/0QH+txXiz4A1FqMceU5tkfJxMI84lgUBmAizoAoWQAnMhZshBDfCVBhi F4Kd+MuoJlP8DJWVpeOCkwqKZwTWBwIcC9jOf2f3L37qtc0BGIhd0zH8n0d4GsZcBfD/CogvQehf AWkjbKRy6+dgy+dBHgfSvwLCISj8HEy6DCKDO2lvm64G6Zewgvo3EhRfBrYL1hD9KJZHryDhfyMn LgeSeUfkcHe9pfAj8AwJ7mlP9Pu87GivvDTYd+kBJaaeI1otrh/0/Td0zlVdCmVuZHN0cmVhbQpl bmRvYmoKMzQgMCBvYmoKNzg2OQplbmRvYmoKMzUgMCBvYmoKPDwvU3VidHlwZS9UeXBlMUMvRmls dGVyL0ZsYXRlRGVjb2RlL0xlbmd0aCAzNiAwIFI+PnN0cmVhbQp4nG1WeVRT17o/MZB9tKitaa6k 1iRLq9Y6VK1eobUOBbRWagFBGVQGDRDNBIQhIQkJJCh+YUhCAgQI8yQgIAiiUmsdih1sq9ZqtQ99 vb2dbH3c9q59WIe33tux6773x3srZ2Vl2N+4f9/v93EovxkUh8MJiJYppFmrolSKZKXv+0pmAYd5 cQazkBvGKqdqp7b4L6TC67mzIYALAX6NL85dMo9Z+hx+NBePPUtxOZyBT74NUam1mbK0dI3k5Zio fctXrFj5v7+sDQ4OlqRo//WPJFSaJUtTSpaSDzlSuUqtkCo1b0hCyGm5XHZIkibXqtOzJMmHD0sP +8z2JsulRyXbZXKZWq3Kkbwcslyybs2atavI27o3JLuzFdJM1UqJTJkqU8o0Wkmy8rDkPYU0LVmi SD4s9TkIVcg0mVrJ+jUy5f9Y75YpUrKzJE9LluxWBUvCJVHStGx5cub//YeiqEXKQ6rDIWppaERY ZmRWepRGln0kemfMO7nyZEXKcsnqtY6XKeo9KpSKoMKol6hIaju1hFpLLaX2UOuoaGon9RoVQ71D 7aV2URuoV6i/UiuoWOpd6i1qNxVCzaI41FxqHkVTM6nF5EIoPyqHusF5gVM149kZFTOecGXch34q vxv+Kf4sL4M3jd5DzfRcOo++OpOaqZ55dlb4rJJnlj9zL2BDQE6ANsAe8PVsevZOXDaHOQ4e5oUP OfcncOgEt5ZZJLCXuWxuoDsq9fFi1oYgymSIPUbrcboHxZbrq6CPxr3o/Nmmqh6gL7TId4pZKYJw oy7Od+iwB8VXFFTBBRpr0U+Jl7cdkOt27RB9gyyl8VqZKTpfOIeJMN9i/Ds5rXe5zCK8RpAQlpYV BfSaXfcx+seHNx/0VZtSXWJ7tlNVr20EYVuTt/ly2LmN8bG5aSniuETVDthEs3/5ZjXmnh2sPdkn am/xtnddo0kl5g5mXQen83tc8z0XlzFJAvYva1awInbBj0vxPDzvl39iEZ7/6s/s82KjSjBxbQW7 kPU/uGPr4dSWwRyx6mL+l3CT/vnc9Zsin6+LzAsdnPZHOHeCix+bBXjZRf9apD1RAseBTityj4qZ T1BxaYIu3bxLJ8xHTpsdnED3V5pTxOwaJO1Wuw8BzaLVrD+7iH3+zno84/JAw7le8S6EOX5gsRSY TdrcTLMS6KDo+3gu9r/w1dcfXNi/Vzxn6lXDCBPUiIMHOR2Y5k5RzF5BIc7xz+FZc/ILcqEQ9GV6 e3xtnHM/BEOQ/N3osHelq4GdAau7Nrwf9dE7P0h/A+wPv42OP6C19dtCdyqCQBgOexqSBqIvyL8H zKXxzh/xTLzkysWcQ4Oi7qMelfcdXwdZRZ2uHkd2M3M+mndnAmc/ms/XVzILBZU2B1QC3esw7xdP pyP+BdhnMkVYaR3i63FxPW+rzeiGj2m8Fp060mI+AzSe9QP2w0vws2F/e2VPYmZMovguspbG5KQb o3VC/MpWQdflvtGvLmxhaZYbvzVkf1Rfl8iHjC+wshujPg6zEgcJErYkKXdBFCR1Kq9oByw9JeM0 HucVfWruzOxW9B1o3AdxsF8rle1PUYdCMM2Kbm3A6Oc71zB1RsTewysF7nu9/eMwDi2q6vX0nKlU 8Ob1Mw/787zzfp9keJgzn1/3O+YIIKTAuKWYzkN8Oe7nYRr+6Ll++dNr/X+Dx/C94l70hzu+YDl9 7EKg+XVscx7vG0tlIWyjJ3sEfPlJt3Po0af5svOiezFDG4ClgJ2VsvH1/bHKyIL1QB8qqOwWz2E+ AC9uGsRjvtBPJnG+L/RjfBCPCNpK6k60wT245Bpu+ai7bxw+gzOGAXXnoZH1HatIwAdsXR7vqyKX GULo6TAEG4vMWyx0Hu7xok1l+hq4TTM5fvzH7Q5HO15wff9Lr0TEsSjXYCvVkH5iX8F4eOhOA94w QEL/Njmfn8n8hN8UqHhxLJWUxS4GWs2Lh+QKi6O4rMQNdTQGvBp9Cx5jdRDdjPUaniXlmN5kMOpU RfsJoFfjTB5f12G3t4vwAh5BWZO+bi3dyuMPH8eL/ac7SLLWagOE0dMRCMIM+ree9lXH9PPm4Efg wU9+5Xw9wXUwLwhqyiptVUC3VuYnitkyBHEWy64CQiEWD9pRpvPAFRpno+7hsXZ7ZbHZLao1uK3V QLd46jp6chpkyUpNaLz4BzKCe7PTjBF5QnK9bFj91GAfpxfzmM2Yx52KZF8TsLyyN0Ze/+Pz6r66 0cGvPhv4+1P0H2BfxC+xe8jrRfYlNnF5V/C1XaLTyd25N7PORQd+vgnPTsKvgYdg3m2j7Tz2+6mr gmEYzTslv5TQvQJYf3hZueTQgdSUfZpoSIDEurTuveeOPgE8E37p/HVodGjkXONVIBMlNDflNzJr G4xN896/joO/JIjDn+MbAv7KY/hFfxOvpKTQcuLEMRBawVxqqiAAc2Znl2cs2BQVG7Kn+9DDBPHV I6dy67JALkw8qoqVyaub8kQ5rYXNxit0Po8fzq6pRg5XeWkVSbWmpNlKHFg72o/VL7j7ydUvx7J6 3usTs9Qnqlp9K7QLB092jF7pkq1rFv1J+wsucW5P4N2E4NbiXwRnRrvqTgM91poWJmbjEewwGyOt f/J+jMvqhC9onID+SL70ekRy1nsHRPgKKi6TFWQW7ckXqov881G5rQwqgB52FiWJp4+Sy7cWRhYR e7kHRdrNVcSeMBt4mX/4hmByEht9Q3B2aplXIC3LdSl72RRsD7xR3VXV097d5j0D52DY2J/RSfPv DkmD2lYt4J9lO/N4d4pcJgh9OgpB5oKtVjIKXV60ucJYDbdoHIcul+JCdtwfb+BNJzCPBPy7HW7n IH7uatKquJzktDzRgYx064oSeg4+SRg+oJvzJaneSQh2KaZ5rhNlVpGl0FpoMUujkxLyj5ksVgsU w3FbSVkJ/QF7ibexK+nahYGWiz0iozsnU2fOBmGqvv1jMT79H4i4JD39eJLzM3EZ4lOfV6P99UQY qkqrbZUgbHkKdD2CELNpu08rzR60s6LI5csbo1icI1BrNCpVk6ajs7mps0PTrBT7dOgqI6gz1My7 OYG3T5B+MVljgmLeARMRIPwL4t9l1+l4149XmSGOZgsQqIoNBpM6W16gAPrw0d4zYv5Z/A271IDO H6/RwwGarUCbzsfe7j9d294uGh72D0LlJWMNfe73a4Qkf+beJAdzSPoxzAFBcVmERmqM0BJxq7B5 Sj02FwibK3Xkcuci2HI805RrNRuKDKAHs0PXqG02VOYDrdHqso8Oai7hGde+xovEsVNb/r+qCC8x v57hYMckFzumKMH0cQR/NZmCCFfgs14UXG6o9hGb8xLCPGjSNrMz6BaeFW/2nx7nWdnN/i28Fsxp 9mI/MmKEVbxTfl4ORsQZ8rH5VmPBxuPE0xjxVJb/lCIJF92uqZ4op73sWB56WFxlIvxNGIMI7ZSX 8/sk/s8RLoOZKgFMVtV+6yLH9HnonrU2G5bRbDOZBYs0X6FO2CXfCqEQ36I+p+229PsE7zaCW273 fQex6MtD31jdJniLZj9F7LYHwXgz5tw59flpUcuZylG462MDwwDzXyTVUJLqSWZMALfd7gd2YjuY h77903b6EDpYkJNhsDpbLSLzqfzaDMLQ2Rp18pDiKqbGb+HnyIKQyuzwcJjPCWz1qNfisoAO9NYs k3otmxm4EWt0owBQLYQmt+Ocg/awiXrUf6zK1BDjyHPk2bVr2ITAl3HNMVeJC1xCaKxy9paTU2/r UXOJ09wUixewPwdWKZwGO5lnZ3VVG1FyS+DfWVNNarnVDkIHVFTWDpJd6rfA+jNlFQ0kxJ/g5/w6 wfVOcQV2nttWZSPbULfLQPaFDAQxBmN4MYG8yYPCywtc8CHN/ISOle7TSc3bdU9XwqtMYMu8/u+2 f4cXP5jP/5HZ/ZlAZygo1hNIWWpGxHgfgovFnfqW7L4kbwTRxburtsWHaxpym1saG5rKT1SccIhL Kk84wUG3dzcNXmhXxoh2I3b1u7qiJCnN/zE71yA78kL4SOL4yFDjxXFRxb76nCHoJcWfGqDZV/9N APLiwpyCLIOqMAfodFXvsLgcwbWhPjx7hADFSyZ74UXOowkc7nu4XrInV5c5fXtyW2V+gpi1+2rU 7zlBasz0oL2lejecpvEpBKVOl8PR2jxcdwrokaZ0Qq0KBHtMpigfA2g8KMqmr4PrNK5AXcM99aNA X6lXB4nZNLJ3FxVFWsghlQdllCpKjfVwWohTEJ51+KPNkbEZkTEi7RVZewwkgsoQFE7fISt2bB5Z sfVCH210M898zPliAud+x8Wrpt4UFJcm69VF+41CpY+r7bZycAA9WFmYLJ6uRfKHR+7jwEk8Ay/G c7f8tGx3ZFp8rmiv3+lzfRfvjb3JzmG5CbuC9x1sbhWRecMJj7gDbJjgq/Ndbf2jN//91FPdw8+n Pn5tMuV8RtO+7oL8wPvh7ekdW1vedm+BZbDYGKTerQpXpIVEkoEtIw9+7olAkZkhV7Zmdp5sa+08 mdGmED9VJQ6WPeLikakgAfv2Gn8Dqqpwl5eX1jvqyl1AN1bqj4qn4xAcKdLkF+oLDceTfS3v9KBN tnwXnKeZxwSKj/7JhAlu9Jwdh4/osfTTB9OzNLKjzXl9dofNZhc5SgFsQDvt1qIjqrSYVDExOE9C nichtyUoMlIPhK5PXwbsTGD5/cu/WzKU0JZ98WhVbeAbH6tOqe9k3TB9DU/gj+pv2693XO/qu3WV npPbwOxtwNENvK5ZD5/pcgYEPKwPmE1R/w06yi4UCmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoK MzU4MwplbmRvYmoKMzcgMCBvYmoKPDwvU3VidHlwZS9UeXBlMUMvRmlsdGVyL0ZsYXRlRGVjb2Rl L0xlbmd0aCAzOCAwIFI+PnN0cmVhbQp4nGNkYGFiYGRk5A7JzE0t1nXKz0kBcbV+SDP+kGH6Icvc 3f3T/6cFqyyDz1xW3m4e5m4elkU/rIS+twl+b+b/3iDAwMzIWDFxsXN+QWVRZnpGiYJGaFC4pra2 DkLE0NLSUiGpEiaj4JJanJmep6AGZJSl5uQX5KbmlVgrOANV5+RkJiuk51QWZBQrJKakpKaAtIUl 5qRmK7hl5mQWFOSXKWg4ayoYGRgY6gIJI7/M3KTSYoWg/NzEPAW/fEsFHwXf1JTM0lxMCQYGBsYI BgZLBiZGRpb13/v4fqZ1Lyla9pNliZDwgp8uPy1Ef/cVsU3pmtzd17W5c3pTd3F3WUNZjHdzY0l2 UYbWdy2JuvmdE7qndy+YPnl9L5DRM7OHY8nvd0Xs6ztmNK6K+x71e7vEb+ffhblpxXWRrZJF3/uW sPtNLJ/avaR74fzVV99v3Cfx4bfZ9Iopxd0NksIJ3VVVTRmdHMIL2rsbuuq6OIq+v1jCntlTMyP6 8m+v7+slbn9nfvydZ+m8q5Mk+crn/0ic/z18Ptsaru8K3GvW8PB8V1jEw8vAAACWNq/KCmVuZHN0 cmVhbQplbmRvYmoKMzggMCBvYmoKNDI3CmVuZG9iagozOSAwIG9iago8PC9MZW5ndGgxIDEyNDcy L0ZpbHRlci9GbGF0ZURlY29kZS9MZW5ndGggNDAgMCBSPj5zdHJlYW0KeJztWnt4VNW1X3ufM4+E PCYhhMljOCdMZoIZYh4QEiCSM3ngIwUCRDqDRidAWvABsUlA0QKiVBx8xPq4bbUaaQWKD05mECc8 JMq111qVeK0W7SufxfoofHJbXxchc397T0C41/q1996/+nF21l77sX57rb32Onv2ngwxIkqldaRQ 85z5pRUkn1IT2YLF17Z1JOrnNxOxGYtXdukfafe9jIbfEtmmfavj29fa3989gcjuJLL8+tvX3PCt hPy4nxLx+5a2ty159Z2CrURl69A4ZSkaMtdkxqHwGOqFS6/tuj4hX2YX+q5ZsbgtUS+ohkz2tW3X d6TekqkQpWWhUV/edm37iH09yLSOFZ1dI/iQ6O/4TnuHv2CJkJ9M5Fhh2U35krZSvuqlfKL44VM0 vCx+WPQJzj/E7FwJGnki9AT9mk1gOkXZcRpLn7McVk4Xk0qfwVM76CTdT1nUQg+wTCqkbLqULmYq ZHx0B3swvjL+AV1A36fN8WfY+vh29N9NP6fPYcHvVUZVNBvyl1I7faC8S8H4j8hOt9Eomk7zWDa1 0ZtIn8CGe+k+epbdFP8cWrNoPcarIT/548/FT1Ax3aH2WA4lPU330B5mjS+OL6NxNJ7C3Bd/M/4H 8lKQfkJPwCYfG1AvogK6mjbQD1iO8nOU7qef0jBL4a1KvWU/NF1MC2g5raIwbaeXWCZrthyyHIvf GH+PrDSaJsCmZfQBq2Sz+GNqSnxG/G26jPrpRcxXpAH1MnWr5bLh2viP48/TGHqGJbO97DlLheWu kzfHH40/RSmwpxwemQ09i+gWeo5+Qf9Bf+Fr42vpIpoPzS8wF9OZFx5/k+fwNXyN8jqdj9m2wtpu eoRMrMhu2kP74Jvf0BC9y7JYHruELWL3sL/wFL6EH1QeVHYqv1KZ+jP4200e+KiLHqNd9DK9QgeZ BeOXsWZ2FVvB/oX9mA1xkx/hn6l29Rb1C/WkxTs8NPxFfHb8E3JSLn2DVtNa+PYnFKWd9Cq9QX+h v9KnzMGq2VL2KDPZEDvCk/h4Pod38Af4Y/xJZbZyj/KcWqnWqVerr6hvW75n2WRrsw2f2DJ87/CT w6/Fn4m/hthJw/hemgmP3oyoeIz20+sY/S36Hb0j4gfjT2cL2RXQ0sk2svvYk+wF9hr7ELMkmcbz 6bwBWlfw78BP6/m9/D5oP4g0yN/mv+N/5p8oFmW8MkW5TnlUMZWYMqj8SXWoXvV8tVydoy5U41iZ CsuFlvmWbZbHLc9bjllrrEusHdb3bettt9pfPll88vfDNLx02ByOInbtiKTV8MTDtBlxvxNr8BI8 +iosHqKPsQq5rIAVwe6pbCZrYrPYN9nlrJ2tZ7ex77MfsAfZZvYUZoA5cBts93E/n8/beDu/ld/G 7+Q7kXbzX/A3+SF+FJaPVdyKTylXLlYWKpcpyzGHLmWNcis8e4+yXTmovK68p7yvHMWqjVXHqd3q avWH6lZ1p/qa5RuWa5E2W/ZbBiyvWU5YTli5Ndeaby21XmXdZn3HZrVNsTXbbrf9yvZXewfLZ8Ww XKczHp6Dd3Ac386z1LXsKBpcTKV0zNyHdZiPt+KvVKsMY13SRD9sG8Nz1NECaTVU7Je8i+2hSvYC rbVyBbuqOkQR9ls+pB7gF9AbLMRy1K3KcstLvIAex27Uw/fyPayOdvIavoA/pBB7l22jdxHv19N9 7GrWSY+zo2wa+y6rYmvpVzxbmc9upZr4Zq6yJHYxO0awgG5Wl9AV9LUPm4rd+oPhh9VU9SbsTzF6 ACv6BP2B/YyOM0v8CHY3BbtRG3aZOxDvG0jseq14z9bifczBDnKN9SDtZFbs+FXWGepqOkb/SR9Y diOi6rCTvje8TH1Y/WO8Kl6CNwxvGW3De7eULsQb8y6iZB/qonY53vRk7CUVeKubaSEtoe9i17sn bsYfit8SvyG+gn4J7HE2kR1nvXgjYkDU0ItId9NbbBPewwu/fp5/6xleQgP0IXMyD6vA+3DUstLS Y9lu2Wl51vKKtRzevpUeRES/g2hOxgwW02v0IX3G7FibHJpIk2FvNWwP0DU8qOyjepZLHXhnJ2Af rxuZSSdGWQ/vPYT3eR/ejWPYJy6nZ+kQ42wsZrQY+u0Ypwl+vhLSW7CCt7AoWpZg1y6mP2Peaaya d0GfgZEewK41AJt+S3+Ct+PSronYFxrYAoz1GX2TlkDDFGpmfViBXTQVO2uD8jL8XcgcVMfGs58C F8IbmkYummr5I+M0cXh2vJovU/bhMyaO9l58euXRBew6WJGOeZykMWwOVQ7Pgw2vExn+FqN2xgU1 06dNra6qnDyporys9PySib7i8yYUeT2F7vEFujbOlZ+Xm+Mcmz0ma3RmhiM9LTVlVHKS3Wa1qApn NLHRPTOkm96QqXrdF11UIuruNjS0ndEQMnU0zTxbxtRDUkw/W9KA5Lf+m6SRkDROSzKHXkM1JRP1 RrduvtLg1mNs4dwAync2uIO6eVSWZ8lyjyynolxQAIDe6FzaoJsspDeaM1cuDTeGGjBc36jkend9 e3LJROpLHoXiKJTMse6OPjZ2BpMFPrZxWh8neyqMMnPdDY1mjrtBWGAqnsa2JWbz3EBjQ15BQbBk osnqF7sXmeSuM9N9UoTqpRrTWm/apBp9mZgNbdL7Jg6E74g5aFHIl7LEvaTt8oCptAWFjgwf9DaY Y1cfdn5ZxeCZ9YHbzuzNU8KNzmW6qIbDt+nmwNzAmb0FIg8GMQaw3DMzFJ4J1XfAiU3zdWjjG4IB k22ASl3MRMwqMb92d6NoCV2lm0nuOvfS8FUhLE1u2KR5NxREcnON/vgQ5Tbq4ZaAu8CszXMH2xry +7IoPO+GaI6h55zdUzKxz5GRcGxfWvpIISX1zEL76T5ZkuKi1DTvtGeZsMh9MQLC1BfrsCTgxpyq RdZeTeHF1RDDE2RAmUuwIsvMpPpQ2DFNtAu8afE43Hr4E0IEuI8eObulbaTF6nF8QqIo4uR0qKH/ VNn0+cziYhEitnqsKWycIeuVJRNXxvgUd4dDB4P7qBm+bQtOK4X7CwrEAm+KGbQIFXPd3ECirtOi vAgZpb6gyUOiZ+BUz5hLRc+6Uz2n4SE3InkniUP9GNPuPf2X7sge3bh0msmyv6a7PdHfNN/dNHdh QG8Mh0Z829RyVi3RX326b6Rkjq4PKHl8pMTzFNmLoLz8tLCoBFJM1YM/qwzqJaaCoJQNTJ9pOkIX JfJgckHB38TEbPYzQLH4MYGS7EvYiJXmNN/Z9eln1c+yLiWswF7Vy5taFobDyWf1zcQGFA7PdOsz w6FwWyy+bpFbd7jD/Xwr3xruaAydWtBYfPemPHPmHUFMYimbhmDlVNfnZhvn9hls4/yFgX4Hbiob WwIRznh9qC7YV4i+QD+OIoZs5adbRU0XNWpiCPQIt8uuvH6DaJ3sVWWDrC+OMZJt9lNtjBbHeKLN IdvwlJBYe9uM4dlU76Djx4e9Dtly1jNGtFjD+PyuoRW4AXByUCk+ucg6Ox7HWYE/Sy3KjyidMdLi A8oPoo6sCiOm/DCaPrrC8DuU+6kZxMlUZtEAiNMK5R5aC+IQb4qUlFf0i0I0Oa3CAflNpIPWgRTq Rc5k3QAJ+U3R0dli+Fsi6RkSd2OkbHKiEHU4K5r9Wcr1xJR2ZTkO/RoOi8vxkaopi8Fd4IuUJbje CjuNaLqjYh301UK8Fmen89DtV7JxItGUBiUXn4ZCrDuSltDTHZlQXOFPVuoVpxRJV1JxGNAUu2KL VGj6HkW42FA2RpNGCfs2RhxjKvYpGxQbLmuasg5SY7X0fUoylYLETFqiSakVPf4UpQXTbIFbNNjI 6BGZG8ryCAaCvkYlHxcYTblacWEpNGWmMi4yRhvYo9wrxb4vRoG+GRH7JMGiqWkVA/4kZQZ6TeUu ePwuqa0n6q3GWcurTKAyEIdT16K0ViynEkYpjGUKY2nCWJowrAhjqUm5HT23Q6ZUWU0dyirqAT2C soohx0TgwX5ZKJxQ0a/kKE54wrEHvmNozY0mpQnLnJHM0VLMGU1Jq6jdp3TSHBCH8V3Rsc6KFXuU YjmViVFnngB0RJJS4LqxibUAMFuswT4lXxknPeGSHjD9GuqM0hWNGH+JDwrv8Nf5G2J9xfVH8l+O 8FdG+KsJHh/gg1FoMWL83wUf8ufzdzHYlfx39AhKnO/hB6gMgLd5TFjB3+L9VAt+CPUl4P3gk8B3 Rwpe1GI8FgWD7Q9GUrPFZPmBiK90pKB5Rgpj80YKmdkVfg9/nj9H+Rji1+CF4M/xAVzZNb4f3Ak+ gAPgi+BP80qaDr5zhP8r3ytimj/Dd+EoqvFoJE2YYEZsgu2IWAV7KkKJWnOptpc/xR/HLVbjT0a8 uWjdFvUWaul7MB7DZbEr4tIy/cn8URZgH0OoFwdVcMrkmyNVYpCeyF5d6+c9vMdwVhkeo8TYopR5 ykrKtii6Ry/Rq/Qtut/B7yILnIcXlm9CXkU6R/SADFAPvz2iVpn+k5iTmBendch7ZSmEvEOWcGki x+neY7JUyzfQHBDHGGtAa0HrQDfjgtLDV4NuBN0E+q5s6QJ1g1Zh++gAogOIDiA6JKIDiA4gOoDo kIgOqb0bJBAhIEJAhIAISUQIiBAQISBCEiHsDQERkohmIJqBaAaiWSKagWgGohmIZoloBqIZiGaJ MIAwgDCAMCTCAMIAwgDCkAgDCAMIQyLKgCgDogyIMokoA6IMiDIgyiSiDIgyIMokQgdCB0IHQpcI HQgdCB0IXSJ0IHQgdIlwAOEAwgGEQyIcQDiAcADhkAiHXJ9ukEAMATEExBAQQxIxBMQQEENADEnE EBBDQAzxVX3KoP8FQAYBGQRkUEIGARkEZBCQQQkZBGQQkMGRqXdJZ3CEzRrQWtA6kMAOADsA7ACw AxI7IMOrGySwJhAmECYQpkSYQJhAmECYEmECYQJhSkQvEL1A9ALRKxG9QPQC0QtEr0T0ysDtBgnE Px6U//DS8JtZwI4PV76OnSf5Wjoi+Ro6JPl3qU/ym2iL5DfSeslXU5Xkq8grOcaTvIs0O4toVen+ bGwBc0BXglaAHgHtAO0H2WTpIOgPoDivNMar6bY5tkdsO2z7bZYdtiEbT7fOsT5i3WHdb7XssA5Z ue7P46lyH8XWQnfLfC3yj0D4EEFeK0u1fDL0TsY+W4k0mU82Mo7qHxWzg8VsfzHbUczuLmb+JH4h U+VOp1MVLpAaCxgp3hnaIVCVt2gGdqa7dh0Zq0W8U7QY25tg5xk+8COgPtAW0HpQFagCVALygDTZ Vgz5gDF+ZMi9oCJQAUgXKig7G8efzAy70c9T2ZboC6mUJPQUTQBuT6SoDCwWKZoD9kykaJHmT2K7 qEgcg9jTWLnHwXdEtMPofjLBnohoe8C2RbTJYK2RovPBLosUvaL5U9mlpKkC2jLC52Pegs+LaAsg NjeinQfmixR5hXQxFHnQex4L0GFwzwiqMKHJHdGmg42PaFOFtJ2KxMIzK5VI8ywgwZUoDPqonwVU ZozSjmr3akcA/zMci/B4S4+pYAc9MbbASNb2ljwMYb8W8ScLeXw+9I1wU/CntS2e27UHMRbz7NJ+ qJ2v3VUSs6P5Tth9u1QR0dbjsvO4MVpbp5VpXSWHtU7tEq1Nm6e1etAe0S7X9gozKcgC/PFdWjMG vBiz8ES0Cz0xaeJM7QbN0Iq0qfpe4V+qToxbVbJXeIAqEtonwr/FnpiI8UurYizDKLYds/XYLrPV 2abb3LbxtnE2ly3Lnml32NPsKfZku91utat2bid7Viw+ZPjEITjLKs/CVlXkqiw7uMg5yTMyZ3ZO l5A5WmniTfPrWJM5sJiaFunmp/PdMZaMu4TFXcfMzCZqaqkzq31NMVt8nlnlazJtzZcF+hi7K4hW k2/ESb0lEGNx0bQhT1za+xhtuDOvnxjL2XBnMEjO7JW1ztrMGRlTZzZ8RRYayX1fPs4ziy7zgab5 AXO7K2hWiELcFWwybxZX+n6ezlMbG/p5mmDBQL/awdMb54l2taMhCLHDUgzRnAYxKhIMYvY60oUY 9pM6IYY1Ssh5AYdcgWCQS04lr5TzJqdKOZUJub5DemNDn65LGQ/RISlzyENnyCBigG3o83qllFtn ASHFAm5dGnaeHEjTIFKiSRGGc50cSGNSmVn6pYhnRKTytEil1KWwL2W0hEzWhFMyWRMg4/s/Pu11 PhYt715zQHxLEnI3toNC5qaVS53mukW63reme+TrE29o0eKlgre1m93u9gZzjbtB7ys/8BXdB0R3 ubuhjw40tgT6DhjtDZFyo7zR3dYQjNbWBPxn6br9tK5AzVcMViMGCwhdtf6v6PaL7lqhyy90+YWu WqNW6mpcJuK+OdBnp7ogbuWSR/moZMRwKK8gWJft6JghArp/eoFzTd5uldg2GuULminuOjMVJLpK /CV+0YX3THSlia/CRrqca6YX5O1m20a6HGjOcNfRKdeSEGoyK+c2mQW4SYtQMY22r16zTvHIbic1 LmvAH+pdkpDOlKTOr3y6vurp7u7uFFm3r5OoySye32ROmQtLbDaoCjUE0Xb+qTZFkW19SUmNsfgA On0wgnUJdaLkYz540EjGrcvGe629Ni6uCl3RXFfFin34BF8Lwj2Or4qUyvsyXxUd7xH3l65oaWWC 434qeCS3oAIaolWACu5JcCOjBIUeT09JT1Wvp7ekt8qK1l1b0KhtER+lkdItCnX5Ok85AsWuIJwN s4S+RyP5Lqm4VxR8vqCvk0l//U9ns1NOP+3YzpFRO+XwXacWJNHeSQnhRKev+xSoewQiO7slRCrE 1ost2IKE05SN6nZyNmy1xXitMZos6rBCyTZ1mFGO3WoZ5spe5qUkZjInOX2OT2tO1sx2fFwz62QN 1aLsOIGsvKwgoyDDgwwbPZ3QlYEThoW+IF0dEDv9ZqjxWgYoiRYYSVfzG3G7UbiKk0b0SguzxPgV z9iTLIxSkmgP9iyO60+rkWohVVN11VRVNSd5N9vKeqEemltrZgkboLy25uPWo1PLy6i1oCDDaquc Ulg1SfEOv/ej15YzXnZYdfc0xgt/8T1hwSQiNQUWuFitceXTzl25/Xkvqf/mHHQO5gzm2uvz6vPr XQtyHlTvd25Xt+Tbrbk6TbBW5V6k1jvrc+pz7YXOwpzCXCXbqy5QNzofynso/yHX9vztLnsmuRwu 3VXuWum61dXjetNld4lIyc4aM9nFHSnpLgfphNNXGRlwtQimzOzJFOOPRjlLSRfnAreWUprCUwy0 p2wZbUk6lJ3N5sDkXC39kGMVzxn3+vNy3rM+Pjrb8el1NTWzHEep9qTvusNwv6/1upqMzKksY5Kv VUQZueIDkYypwoZIumRGmmOqandMtdgzwDOmJgIj2Gfl9S0BY1RSXk4ezxvNxP/dMBD+WoPlZay1 aW5gH+XFhygf5IoPVVdXB9l1ra2tLKNgSmbVlKoplZO97vFWm2dK4aSK7DFZVptVtdrUlBNFjt4j z/qmtQcDS+3D7+cw+8/f+vzCWZOGP70wm1mGv7iPJf2mr/abl17RftWN+e+/9OFTi6OL/B83e8X3 ZFP+zrSKXXIunUvn0rl0Lp1L59K5dC6dS+fSufTPlhI/pBz5RUEWrpDiq9NckFXUx/zvfsD3z/Ko VChzVfjnGI/HkTORi++bSfgr4SBGmfKnvkRWSiVqaWksr6+smDKpuqusTH4TzXrEVyF/52M/u3qM jsXPahj59YdVfPX9/0hqJ20W32MkYuBrHqGf/+nbfeaO3Vem13xiz0kYvPmPNfLX+P1vPD36+PET Jx1kF79LSTpl738BifF6kQplbmRzdHJlYW0KZW5kb2JqCjQwIDAgb2JqCjU4NDUKZW5kb2JqCjE3 IDAgb2JqCjw8L0Jhc2VGb250L1ZMUkZXSCtUVEUxRjY4Q0IwdDAwL0ZvbnREZXNjcmlwdG9yIDE2 IDAgUi9UeXBlL0ZvbnQKL0ZpcnN0Q2hhciAxL0xhc3RDaGFyIDE2L1dpZHRoc1sgNTkyIDUwMSAy MjggNTkyIDY4MyA1OTIgNDU2IDI3MyAzMTkgNDU2IDUwMSA0NTYgNzI5IDIyOCA3NzQKNTAxXQov RW5jb2RpbmcgNDEgMCBSL1N1YnR5cGUvVHJ1ZVR5cGU+PgplbmRvYmoKNDEgMCBvYmoKPDwvVHlw ZS9FbmNvZGluZy9CYXNlRW5jb2RpbmcvV2luQW5zaUVuY29kaW5nL0RpZmZlcmVuY2VzWwoxL04v by9zcGFjZS9DL00vUi9lL3Qvci9hL24vcy9tL2kvVy9oXT4+CmVuZG9iagoxOSAwIG9iago8PC9C YXNlRm9udC9RVUdWTFArVGltZXMtUm9tYW4vRm9udERlc2NyaXB0b3IgMTggMCBSL1R5cGUvRm9u dAovRmlyc3RDaGFyIDMyL0xhc3RDaGFyIDE0Ni9XaWR0aHNbCjI1MCAwIDAgMCAwIDAgMCAwIDMz MyAzMzMgMCAwIDAgMCAyNTAgMAowIDUwMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAKMCAw IDAgNjY3IDcyMiA2MTEgMCAwIDAgMzMzIDM4OSAwIDAgMCAwIDAKNTU2IDcyMiA2NjcgMCA2MTEg NzIyIDAgMCAwIDAgMCAwIDAgMCAwIDAKMCA0NDQgNTAwIDQ0NCA1MDAgNDQ0IDAgMCA1MDAgMjc4 IDI3OCAwIDI3OCA3NzggNTAwIDUwMAo1MDAgMCAzMzMgMzg5IDI3OCA1MDAgMCA3MjIgMCAwIDAg MCAwIDAgMCAwCjAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAKMCAwIDMzM10KL0VuY29k aW5nL1dpbkFuc2lFbmNvZGluZy9TdWJ0eXBlL1R5cGUxPj4KZW5kb2JqCjI4IDAgb2JqCjw8L0Jh c2VGb250L1lZU0lUWStUaW1lcy1Cb2xkL0ZvbnREZXNjcmlwdG9yIDI3IDAgUi9UeXBlL0ZvbnQK L0ZpcnN0Q2hhciA4OC9MYXN0Q2hhciA4OC9XaWR0aHNbIDcyMl0KL0VuY29kaW5nL1dpbkFuc2lF bmNvZGluZy9TdWJ0eXBlL1R5cGUxPj4KZW5kb2JqCjE1IDAgb2JqCjw8L0Jhc2VGb250L01VSE9T WStUVEUxQzUyNjM4dDAwL0ZvbnREZXNjcmlwdG9yIDE0IDAgUi9UeXBlL0ZvbnQKL0ZpcnN0Q2hh ciAxL0xhc3RDaGFyIDIvV2lkdGhzWyA1NTYgNTU2XQovRW5jb2RpbmcgNDIgMCBSL1N1YnR5cGUv VHJ1ZVR5cGU+PgplbmRvYmoKNDIgMCBvYmoKPDwvVHlwZS9FbmNvZGluZy9CYXNlRW5jb2Rpbmcv V2luQW5zaUVuY29kaW5nL0RpZmZlcmVuY2VzWwoxL29uZS90d29dPj4KZW5kb2JqCjE2IDAgb2Jq Cjw8L1R5cGUvRm9udERlc2NyaXB0b3IvRm9udE5hbWUvVkxSRldIK1RURTFGNjhDQjB0MDAvRm9u dEJCb3hbMCAtMTIgNzcyIDcyOF0vRmxhZ3MgNAovQXNjZW50IDcyOAovQ2FwSGVpZ2h0IDcyOAov RGVzY2VudCAtMTIKL0l0YWxpY0FuZ2xlIDAKL1N0ZW1WIDExNQovTWlzc2luZ1dpZHRoIDIyOAov Rm9udEZpbGUyIDMzIDAgUj4+CmVuZG9iagoxOCAwIG9iago8PC9UeXBlL0ZvbnREZXNjcmlwdG9y L0ZvbnROYW1lL1FVR1ZMUCtUaW1lcy1Sb21hbi9Gb250QkJveFstNzAgLTIxOCA3NzUgNjgzXS9G bGFncyA0Ci9Bc2NlbnQgNjgzCi9DYXBIZWlnaHQgNjgzCi9EZXNjZW50IC0yMTgKL0l0YWxpY0Fu Z2xlIDAKL1N0ZW1WIDExNgovTWlzc2luZ1dpZHRoIDI1MAovQ2hhclNldCgvbi9jL28vZC9DL3Av ZS9EL1AvRS9yL1Evcy9oL1IvdC9pL3Uvai9UL0kvVS9KL3cvbC9hL20vYi9wYXJlbnJpZ2h0L3Nw YWNlL3BlcmlvZC9vbmUvcXVvdGVyaWdodC9wYXJlbmxlZnQpL0ZvbnRGaWxlMyAzNSAwIFI+Pgpl bmRvYmoKMjcgMCBvYmoKPDwvVHlwZS9Gb250RGVzY3JpcHRvci9Gb250TmFtZS9ZWVNJVFkrVGlt ZXMtQm9sZC9Gb250QkJveFswIDAgNjk5IDY3Nl0vRmxhZ3MgNAovQXNjZW50IDY3NgovQ2FwSGVp Z2h0IDY3NgovRGVzY2VudCAwCi9JdGFsaWNBbmdsZSAwCi9TdGVtViAxMDQKL01pc3NpbmdXaWR0 aCAyNTAKL0NoYXJTZXQoL1gpL0ZvbnRGaWxlMyAzNyAwIFI+PgplbmRvYmoKMTQgMCBvYmoKPDwv VHlwZS9Gb250RGVzY3JpcHRvci9Gb250TmFtZS9NVUhPU1krVFRFMUM1MjYzOHQwMC9Gb250QkJv eFsyNCAwIDYyNSA3MThdL0ZsYWdzIDQKL0FzY2VudCA3MTgKL0NhcEhlaWdodCA3MTgKL0Rlc2Nl bnQgMAovSXRhbGljQW5nbGUgMAovU3RlbVYgOTMKL01pc3NpbmdXaWR0aCA3NTAKL0ZvbnRGaWxl MiAzOSAwIFI+PgplbmRvYmoKMiAwIG9iago8PC9Qcm9kdWNlcihHUEwgR2hvc3RzY3JpcHQgOC4x NSkKL0NyZWF0aW9uRGF0ZShEOjIwMDUxMDIwMDkyNjEzKQovTW9kRGF0ZShEOjIwMDUxMDIwMDky NjEzKQovVGl0bGUoTWljcm9zb2Z0IFBvd2VyUG9pbnQgLSBhY3RpdmUgYWN0aXZlLnBwdCkKL0Ny ZWF0b3IoUFNjcmlwdDUuZGxsIFZlcnNpb24gNS4yKQovQXV0aG9yKGdkcm9yKT4+ZW5kb2JqCnhy ZWYKMCA0MwowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDQxMTEgMDAwMDAgbiAKMDAwMDAzNjI3 MCAwMDAwMCBuIAowMDAwMDA0MDM1IDAwMDAwIG4gCjAwMDAwMDM2MDMgMDAwMDAgbiAKMDAwMDAw MDAxNSAwMDAwMCBuIAowMDAwMDAxNDk2IDAwMDAwIG4gCjAwMDAwMDQxNTkgMDAwMDAgbiAKMDAw MDAwNDIwMCAwMDAwMCBuIAowMDAwMDE1Mjc2IDAwMDAwIG4gCjAwMDAwMDQ3NjMgMDAwMDAgbiAK MDAwMDAxMzM0NyAwMDAwMCBuIAowMDAwMDA2MzY2IDAwMDAwIG4gCjAwMDAwMDY2NDAgMDAwMDAg biAKMDAwMDAzNjA2OSAwMDAwMCBuIAowMDAwMDM1MDg2IDAwMDAwIG4gCjAwMDAwMzUzMzIgMDAw MDAgbiAKMDAwMDAzNDE0MSAwMDAwMCBuIAowMDAwMDM1NTM3IDAwMDAwIG4gCjAwMDAwMzQ0NzIg MDAwMDAgbiAKMDAwMDAwNjUwNiAwMDAwMCBuIAowMDAwMDA2NTU4IDAwMDAwIG4gCjAwMDAwMDY1 ODggMDAwMDAgbiAKMDAwMDAxNTczOSAwMDAwMCBuIAowMDAwMDAzODE4IDAwMDAwIG4gCjAwMDAw MDE1MTYgMDAwMDAgbiAKMDAwMDAwMzU4MiAwMDAwMCBuIAowMDAwMDM1ODU5IDAwMDAwIG4gCjAw MDAwMzQ5MjcgMDAwMDAgbiAKMDAwMDAxNTc5MyAwMDAwMCBuIAowMDAwMDE1ODQ1IDAwMDAwIG4g CjAwMDAwMTU4NzUgMDAwMDAgbiAKMDAwMDAxNTkyNyAwMDAwMCBuIAowMDAwMDE1OTkyIDAwMDAw IG4gCjAwMDAwMjM5NDYgMDAwMDAgbiAKMDAwMDAyMzk2NyAwMDAwMCBuIAowMDAwMDI3NjM2IDAw MDAwIG4gCjAwMDAwMjc2NTcgMDAwMDAgbiAKMDAwMDAyODE3MCAwMDAwMCBuIAowMDAwMDI4MTkw IDAwMDAwIG4gCjAwMDAwMzQxMjAgMDAwMDAgbiAKMDAwMDAzNDM1NiAwMDAwMCBuIAowMDAwMDM1 MjQ0IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgNDMgL1Jvb3QgMSAwIFIgL0luZm8gMiAwIFIK L0lEIFso/upVch1IqlhptZDi5KszuSko/upVch1IqlhptZDi5KszuSldCj4+CnN0YXJ0eHJlZgoz NjQ3OAolJUVPRgo= ------_=_NextPart_000_01C5D549.65D8E2EE Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib ------_=_NextPart_000_01C5D549.65D8E2EE-- From ipoverib-bounces@ietf.org Thu Oct 20 03:35:54 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxu-00086n-6M; Thu, 20 Oct 2005 03:35:54 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxq-000865-EV for ipoverib@megatron.ietf.org; Thu, 20 Oct 2005 03:35:52 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA21624 for ; Thu, 20 Oct 2005 03:35:39 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESV9i-0005bz-9a for ipoverib@ietf.org; Thu, 20 Oct 2005 03:48:07 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZTN9>; Thu, 20 Oct 2005 09:34:14 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA59@mtlexch01.mtl.com> From: Dror Goldenberg To: "H.K. Jerry Chu" , ipoverib@ietf.org, gdror@mellanox.co.il Subject: RE: [Ipoverib] ipoib-cm nits/clarifications Date: Thu, 20 Oct 2005 09:37:14 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.5 (/) X-Scan-Signature: b1c41982e167b872076d0018e4e1dc3c Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0183684278==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0183684278== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D549.65CF5986" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D549.65CF5986 Content-Type: text/plain > From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com] > Sent: Wednesday, October 19, 2005 6:57 PM > > >- page 10, section 6.0 - what is the final MTU of a > connection ? is it > >the minimum between > > the two "receive MTU" , or is it hybrid - different MTU per half > >connection ? > > Although there is really no reason to demand for a single MTU > value per connection as long as a sender never sends a > message large than the "Receive MTU" of the receiver, I'd > expect most implementations would want to choose one value > for both directions for simplicity. For that I'd suggest to > take the min as the final MTU for both directions so that the > side that advertises a 32KB MTU doesn't need to post 32KB > receive buffers and waste memory if the other side will never > send msgs larger than 16KB, for example. I agree. > > >I think that the current model for multiple connections > establishment > >between two peers works well for 1 connection but is broken for more > >than one connection. > >Because of cases where CM packets are dropped or travel on > different VLs, > >they > >might get reordered on the way and observed in a different > order than were > >originally > >sent. This breaks the current model which assumes that the > REQs are sent and > >processed in order. > > Isn't this a problem at the IB layer already, or this problem > is introduced by the 20byte comparison for the REQ crossing case? I think that it's a new problem because we're not letting the CM perform Active/Active on behalf of the ULP. However I think that the fundamental issue here is also that we're trying to use both Active/Active and Active/Passive semantics on the same "service" while expecting to end up with more than one simultaneous connection. Take a look at the pdf I've sent in the other mail, it tries to illustrate the problem. -Dror ------_=_NextPart_001_01C5D549.65CF5986 Content-Type: text/html RE: [Ipoverib] ipoib-cm nits/clarifications

> From: H.K. Jerry Chu [mailto:Jerry.Chu@eng.sun.com]
> Sent: Wednesday, October 19, 2005 6:57 PM
>
> >- page 10, section 6.0 - what is the final MTU of a
> connection ? is it
> >the minimum between
> >   the two "receive MTU" , or is it hybrid - different MTU per half
> >connection ?
>
> Although there is really no reason to demand for a single MTU
> value per connection as long as a sender never sends a
> message large than the "Receive MTU" of the receiver, I'd
> expect most implementations would want to choose one value
> for both directions for simplicity. For that I'd suggest to
> take the min as the final MTU for both directions so that the
> side that advertises a 32KB MTU doesn't need to post 32KB
> receive buffers and waste memory if the other side will never
> send msgs larger than 16KB, for example.


I agree.

>
> >I think that the current model for multiple connections
> establishment
> >between two peers works well for 1 connection but is broken for more
> >than one connection.
> >Because of cases where CM packets are dropped or travel on
> different VLs,
> >they
> >might get reordered on the way and observed in a different
> order than were
> >originally
> >sent. This breaks the current model which assumes that the
> REQs are sent and
> >processed in order.
>
> Isn't this a problem at the IB layer already, or this problem
> is introduced by the 20byte comparison for the REQ crossing case?

I think that it's a new problem because we're not letting the CM
perform Active/Active on behalf of the ULP. However I think that the
fundamental issue here is also that we're trying to use both
Active/Active and Active/Passive semantics on the same "service" while
expecting to end up with more than one simultaneous connection.
Take a look at the pdf I've sent in the other mail, it tries to illustrate the
problem.

-Dror

------_=_NextPart_001_01C5D549.65CF5986-- --===============0183684278== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0183684278==-- From ipoverib-bounces@ietf.org Thu Oct 20 03:35:54 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxu-00087J-Pv; Thu, 20 Oct 2005 03:35:54 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESUxq-000866-EU for ipoverib@megatron.ietf.org; Thu, 20 Oct 2005 03:35:53 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA21622 for ; Thu, 20 Oct 2005 03:35:39 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESV9i-0005c6-9b for ipoverib@ietf.org; Thu, 20 Oct 2005 03:48:07 -0400 Received: by MTLEX01 with Internet Mail Service (5.5.2653.19) id <4XZDZTN0>; Thu, 20 Oct 2005 09:34:16 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA5B@mtlexch01.mtl.com> From: Dror Goldenberg To: Vivek Kashyap , Dror Goldenberg Subject: RE: [Ipoverib] ipoib-cm nits/clarifications Date: Thu, 20 Oct 2005 09:39:26 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.8 (/) X-Scan-Signature: 25620135586de10c627e3628c432b04a Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0697926306==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0697926306== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D549.65DDA7A2" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D549.65DDA7A2 Content-Type: text/plain > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] > Sent: Wednesday, October 19, 2005 7:21 PM > > - page 10, section 6.0 - what is the final MTU of a > connection ? is it > > the minimum between > > the two "receive MTU" , or is it hybrid - different MTU per half > > connection ? > > The receive MTU gives you the peer's minimum value. An > implementation can work this way. However, it might be useful > to state it as the minimum of the two in the draft..since an > implementation is likely to use the > the same size buffers when sending or receiving. That is if A > advt. recv of > 32K it will very likely send packets which are only 32K even > if the peer > send a receive MTU of 64K. > > thoughts? Sounds good to me. ------_=_NextPart_001_01C5D549.65DDA7A2 Content-Type: text/html RE: [Ipoverib] ipoib-cm nits/clarifications

> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Wednesday, October 19, 2005 7:21 PM

> > - page 10, section 6.0 - what is the final MTU of a
> connection ? is it
> > the minimum between
> >   the two "receive MTU" , or is it hybrid - different MTU per half
> > connection ?
>
> The receive MTU gives you the peer's minimum value. An
> implementation can work this way. However, it might be useful
> to state it as the minimum of the two in the draft..since an
> implementation is likely to use the
> the same size buffers when sending or receiving. That is if A
> advt. recv of
> 32K it will very likely send packets which are only 32K even
> if the peer
> send a receive MTU of 64K.
>
> thoughts?

Sounds good to me.

------_=_NextPart_001_01C5D549.65DDA7A2-- --===============0697926306== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0697926306==-- From ipoverib-bounces@ietf.org Thu Oct 20 13:27:03 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESeBy-0002CO-W6; Thu, 20 Oct 2005 13:27:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ESeBu-0002CG-Nn for ipoverib@megatron.ietf.org; Thu, 20 Oct 2005 13:27:01 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA01906 for ; Thu, 20 Oct 2005 13:26:48 -0400 (EDT) Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ESeNr-0001lZ-P6 for ipoverib@ietf.org; Thu, 20 Oct 2005 13:39:21 -0400 Received: from tsx2.cup.hp.com (tsx2.cup.hp.com [15.13.185.29]) by palrel10.hp.com (Postfix) with ESMTP id 9DDE9313C; Thu, 20 Oct 2005 10:25:05 -0700 (PDT) Received: from expvr082905.hp.com (vr724251.cup.hp.com [15.13.108.169]) by tsx2.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id KAA12405; Thu, 20 Oct 2005 10:25:02 -0700 (PDT) Message-Id: <6.2.3.4.0.20051020102136.02dfe338@tsx2.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Thu, 20 Oct 2005 10:25:01 -0700 To: Dror Goldenberg , Dror Goldenberg , "'ipoverib@ietf.org'" From: Vandana Rao Subject: RE: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA5A@mtlexch01.mtl.com> References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA5A@mtlexch01.mtl.com> Mime-Version: 1.0 X-Spam-Score: 0.1 (/) X-Scan-Signature: 2a9ffb6f997442a3b543bcdaf483b990 Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0909735999==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org --===============0909735999== Content-Type: multipart/alternative; boundary="=====================_178945409==.ALT" --=====================_178945409==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Hi Dror, I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.). So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place. -Vandana At 12:39 AM 10/20/2005, Dror Goldenberg wrote: >After giving it some more thought, I think that there is still a >problem. Please look at the attached pdf. > >In the first slide, everything happens smoothly, the left peer knows >that it has to accept the connection and the right peer knows that >it has to decline. >On the 2nd slide, the left peer REQ transmission is dropped in the >fabric. It is retransmitted independently by the CM way after a >channel has already been established. Now the right side can not >really tell whether this is a new channel request or an old >"simultaneous connection channel". This may lead to two channels >being opened mistakenly. > >I don't believe it can be solved by looking at the communication ID. >This number is just being fabricated by the sender of the REQ and is >just a "random number" with respect to what we're trying to determine. > >-Dror > >-----Original Message----- >From: Dror Goldenberg >Sent: Wednesday, October 19, 2005 9:37 PM >To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org >Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >Hi Vandana, > >Thanks for clarifying this. I agree that if IPoiB-CM implementation >looks at the Communication ID it can make those decisions correctly. >The question is whether this value is exposed through common CM >APIs. For all other purposes, the CM doesn't need to expose the >Communication ID to the ULP. However, seems like it's not a big deal >to expose this value to the ULP. >Anyway, it worth noting this explanation in the RFC so that it's >clear that a proper implementation of IPoIB-CM should monitor >Communication ID as part of the simultaneous connection decision. > >-Dror >-----Original Message----- >From: Vandana Rao [mailto:vandana.rao@hp.com] >Sent: Wednesday, October 19, 2005 6:40 PM >To: Dror Goldenberg; ipoverib@ietf.org >Subject: Re: [Ipoverib] ipoib-cm multiple connection model > >Hi Dror, >The purpose of the MAD transaction ID is to allow one to do what you >say below, i.e. have multiple connection establishment MADs between >2 peers. It is essentially the same as the "channel ID" that you >talk about below. Its just that it is a generic channel ID for any >MAD transactions, not just CM MADs. >I do not believe there is any issue with existing IB CM in this regard. >-Vandana > >At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >>I think that the current model for multiple connections >>establishment between >>two peers works well for 1 connection but is broken for more than >>one connection. >>Because of cases where CM packets are dropped or travel on >>different VLs, they >>might get reordered on the way and observed in a different order >>than were originally >>sent. This breaks the current model which assumes that the REQs are sent and >>processed in order. >> >>I think that we need to think of a model which is robust to CM >>packets reordering >>which can be one of: >>- Adding a "channel ID" in the private data. The field goes 0,1,2 >>and on. It helps >> synchronizing both peers on which is the channel at question >> now. The rest >> of the decision (per channel) of whether to accept or reject >> can be the same >> as defined in the draft. >>- Not supporting multiple connections between the same peers. >> >>-Dror >>_______________________________________________ >>IPoverIB mailing list >>IPoverIB@ietf.org >>https://www1.ietf.org/mailman/listinfo/ipoverib > --=====================_178945409==.ALT Content-Type: text/html; charset="us-ascii" Hi Dror,
I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.).
 So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place.
-Vandana

At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.
 
In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.
 
I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.
 
-Dror
 
-----Original Message-----
From: Dror Goldenberg
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Vandana,
 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.
 
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

--=====================_178945409==.ALT-- --===============0909735999== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0909735999==-- From ipoverib-bounces@ietf.org Fri Oct 21 23:33:20 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ETA8G-0003Ue-2N; Fri, 21 Oct 2005 23:33:20 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1ETA8B-0003Dm-CW for ipoverib@megatron.ietf.org; Fri, 21 Oct 2005 23:33:17 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA14896 for ; Fri, 21 Oct 2005 23:33:03 -0400 (EDT) Received: from iron.pdx.net ([207.149.241.18]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1ETAKQ-0004w5-23 for ipoverib@ietf.org; Fri, 21 Oct 2005 23:45:55 -0400 Received: (qmail 18667 invoked from network); 21 Oct 2005 20:32:57 -0700 Received: from sub25-177.member.dsl-only.net (HELO ?192.168.1.103?) (63.105.25.177) by iron.pdx.net with SMTP; 21 Oct 2005 20:32:57 -0700 Message-ID: <4359B2E8.5060903@strahm.net> Date: Fri, 21 Oct 2005 20:32:56 -0700 From: Bill Strahm User-Agent: Mozilla Thunderbird 1.0.7 (Windows/20050923) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Dror Goldenberg Subject: Re: [Ipoverib] ipoib-cm multiple connection model References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Scan-Signature: f4c2cf0bccc868e4cc88dace71fb3f44 Content-Transfer-Encoding: 7bit Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Let me try to understand what you are worried about: I am assuming that each TCP connection (5-tuple) can be on its own CM connection, and not multiplex a single TCP connection over multiple CM connections. Is this your understanding ? Now if this is what we are talking about - we don't have a problem. TCP only guarantees in order delivery to a single connection - if I have multiple TCP connections to the same hosts, IP packets can be completely reordered/jumbled by the intermediate network - and all I am guaranteed is that each TCP stream will have in order delivery relative to itself. I am not aware of any applications where in order IP packet delivery is required between multiple streams. If this is the case - it has to be handled above TCP anyway, as TCP does not make this guarantee. Comments ? Bill Dror Goldenberg wrote: > I think that the current model for multiple connections establishment > between > two peers works well for 1 connection but is broken for more than one > connection. > Because of cases where CM packets are dropped or travel on different > VLs, they > might get reordered on the way and observed in a different order than > were originally > sent. This breaks the current model which assumes that the REQs are > sent and > processed in order. > > I think that we need to think of a model which is robust to CM packets > reordering > which can be one of: > - Adding a "channel ID" in the private data. The field goes 0,1,2 and > on. It helps > synchronizing both peers on which is the channel at question now. > The rest > of the decision (per channel) of whether to accept or reject can be > the same > as defined in the draft. > - Not supporting multiple connections between the same peers. > > -Dror > >------------------------------------------------------------------------ > >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib > > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Mon Oct 24 12:47:39 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EU5U2-0005tx-UQ; Mon, 24 Oct 2005 12:47:38 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EU5Ty-0005tk-0z for ipoverib@megatron.ietf.org; Mon, 24 Oct 2005 12:47:34 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19914 for ; Mon, 24 Oct 2005 12:47:20 -0400 (EDT) Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EU5gi-0001ly-KI for ipoverib@ietf.org; Mon, 24 Oct 2005 13:00:47 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel10.hp.com (Postfix) with ESMTP id 3E28B346; Mon, 24 Oct 2005 09:47:27 -0700 (PDT) Received: from MK73191c.cup.hp.com ([15.244.201.83]) by esmail.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id JAA09489; Mon, 24 Oct 2005 09:38:53 -0700 (PDT) Message-Id: <6.2.0.14.2.20051024094318.024fe9f0@esmail.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.0.14 Date: Mon, 24 Oct 2005 09:46:07 -0700 To: Dror Goldenberg From: Michael Krause Subject: Re: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <4359B2E8.5060903@strahm.net> References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA4E@mtlexch01.mtl.com> <4359B2E8.5060903@strahm.net> Mime-Version: 1.0 X-Spam-Score: 0.0 (/) X-Scan-Signature: e472ca43d56132790a46d9eefd95f0a5 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1666386800==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org --===============1666386800== Content-Type: multipart/alternative; boundary="=====================_219106037==.ALT" --=====================_219106037==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed At 08:32 PM 10/21/2005, Bill Strahm wrote: >Let me try to understand what you are worried about: >I am assuming that each TCP connection (5-tuple) can be on its own CM >connection, and not multiplex a single TCP connection over multiple CM >connections. > >Is this your understanding ? > >Now if this is what we are talking about - we don't have a problem. TCP >only guarantees in order delivery to a single connection - if I have >multiple TCP connections to the same hosts, IP packets can be completely >reordered/jumbled by the intermediate network - and all I am guaranteed >is that each TCP stream will have in order delivery relative to itself. > >I am not aware of any applications where in order IP packet delivery is >required between multiple streams. If this is the case - it has to be >handled above TCP anyway, as TCP does not make this guarantee. >Comments ? Your understanding is correct. IP packets can arrive in any order on any interface though all implementations prefer them to arrive close to in order and on the same interface (Ethernet aggregation technology attempts to maintain the same interface is used consistently). Now, having multiple IB connections between endnode pairs is not an issue - does not matter what VL, path, etc. is taken. However, ideally, the IP datagrams for a given connection are injected for a given tuple on the same path consistently as this leads to optimal performance - it is not a correctness issue though as ordering is the TCP layer's responsibility. Mike >Bill > >Dror Goldenberg wrote: > >>I think that the current model for multiple connections establishment between >>two peers works well for 1 connection but is broken for more than one >>connection. >>Because of cases where CM packets are dropped or travel on different VLs, >>they >>might get reordered on the way and observed in a different order than >>were originally >>sent. This breaks the current model which assumes that the REQs are sent and >>processed in order. >> >>I think that we need to think of a model which is robust to CM packets >>reordering >>which can be one of: >>- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. >>It helps >> synchronizing both peers on which is the channel at question now. The >> rest >> of the decision (per channel) of whether to accept or reject can be >> the same >> as defined in the draft. >>- Not supporting multiple connections between the same peers. >> >>-Dror >> >>------------------------------------------------------------------------ >> >>_______________________________________________ >>IPoverIB mailing list >>IPoverIB@ietf.org >>https://www1.ietf.org/mailman/listinfo/ipoverib >> > > > >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib --=====================_219106037==.ALT Content-Type: text/html; charset="us-ascii" At 08:32 PM 10/21/2005, Bill Strahm wrote:
Let me try to understand what you are worried about:
I am assuming that each TCP connection (5-tuple) can be on its own CM
connection, and not multiplex a single TCP connection over multiple CM
connections.

Is this your understanding ?

Now if this is what we are talking about - we don't have a problem.  TCP
only guarantees in order delivery to a single connection - if I have
multiple TCP connections to the same hosts, IP packets can be completely
reordered/jumbled by the intermediate network - and all I am guaranteed
is that each TCP stream will have in order delivery relative to itself.

I am not aware of any applications where in order IP packet delivery is
required between multiple streams.  If this is the case - it has to be
handled above TCP anyway, as TCP does not make this guarantee.

Comments ?

Your understanding is correct.   IP packets can arrive in any order on any interface though all implementations prefer them to arrive close to in order and on the same interface (Ethernet aggregation technology attempts to maintain the same interface is used consistently).  Now, having multiple IB connections between endnode pairs is not an issue - does not matter what VL, path, etc. is taken.  However, ideally, the IP datagrams for a given connection are injected for a given tuple on the same path consistently as this leads to optimal performance - it is not a correctness issue though as ordering is the TCP layer's responsibility.

Mike

Bill

Dror Goldenberg wrote:

I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror

------------------------------------------------------------------------

_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
 



_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
--=====================_219106037==.ALT-- --===============1666386800== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1666386800==-- From ipoverib-bounces@ietf.org Mon Oct 24 13:42:23 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EU6L1-0006ic-0E; Mon, 24 Oct 2005 13:42:23 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EU6Kz-0006iN-S5 for ipoverib@megatron.ietf.org; Mon, 24 Oct 2005 13:42:21 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22648 for ; Mon, 24 Oct 2005 13:42:05 -0400 (EDT) Received: from nwkea-mail-2.sun.com ([192.18.42.14]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EU6Xg-0003Xs-GC for ipoverib@ietf.org; Mon, 24 Oct 2005 13:55:33 -0400 Received: from jurassic.eng.sun.com ([129.146.56.36]) by nwkea-mail-2.sun.com (8.12.10/8.12.9) with ESMTP id j9OHfp4u024556; Mon, 24 Oct 2005 10:41:51 -0700 (PDT) Received: from taipei (taipei.SFBay.Sun.COM [129.146.106.178]) by jurassic.eng.sun.com (8.13.5+Sun/8.13.5) with SMTP id j9OHfpHu274157; Mon, 24 Oct 2005 10:41:51 -0700 (PDT) Message-Id: <200510241741.j9OHfpHu274157@jurassic.eng.sun.com> Date: Mon, 24 Oct 2005 10:40:36 -0700 (PDT) From: "H.K. Jerry Chu" Subject: Re: [Ipoverib] ipoib-cm multiple connection model To: krause@cup.hp.com, bill@strahm.net MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: 9CV/UpbAP6uuzc8e0Pqoaw== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.6 SunOS 5.10 sun4u sparc X-Spam-Score: 0.0 (/) X-Scan-Signature: 3002fc2e661cd7f114cb6bae92fe88f1 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "H.K. Jerry Chu" List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org I think Dror and Vandana are talking about connection "establishment", not how to use multiple connections after they have been established. It looks like there are some robustness issues wrt the handshake for setting up multiple IB connections. But I may be wrong of course. Jerry >At 08:32 PM 10/21/2005, Bill Strahm wrote: >>Let me try to understand what you are worried about: >>I am assuming that each TCP connection (5-tuple) can be on its own CM >>connection, and not multiplex a single TCP connection over multiple CM >>connections. >> >>Is this your understanding ? >> >>Now if this is what we are talking about - we don't have a problem. TCP >>only guarantees in order delivery to a single connection - if I have >>multiple TCP connections to the same hosts, IP packets can be completely >>reordered/jumbled by the intermediate network - and all I am guaranteed >>is that each TCP stream will have in order delivery relative to itself. >> >>I am not aware of any applications where in order IP packet delivery is >>required between multiple streams. If this is the case - it has to be >>handled above TCP anyway, as TCP does not make this guarantee. > >>Comments ? > >Your understanding is correct. IP packets can arrive in any order on any >interface though all implementations prefer them to arrive close to in >order and on the same interface (Ethernet aggregation technology attempts >to maintain the same interface is used consistently). Now, having multiple >IB connections between endnode pairs is not an issue - does not matter what >VL, path, etc. is taken. However, ideally, the IP datagrams for a given >connection are injected for a given tuple on the same path consistently as >this leads to optimal performance - it is not a correctness issue though as >ordering is the TCP layer's responsibility. > >Mike > >>Bill >> >>Dror Goldenberg wrote: >> >>>I think that the current model for multiple connections establishment between >>>two peers works well for 1 connection but is broken for more than one >>>connection. >>>Because of cases where CM packets are dropped or travel on different VLs, >>>they >>>might get reordered on the way and observed in a different order than >>>were originally >>>sent. This breaks the current model which assumes that the REQs are sent and >>>processed in order. >>> >>>I think that we need to think of a model which is robust to CM packets >>>reordering >>>which can be one of: >>>- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. >>>It helps >>> synchronizing both peers on which is the channel at question now. The >>> rest >>> of the decision (per channel) of whether to accept or reject can be >>> the same >>> as defined in the draft. >>>- Not supporting multiple connections between the same peers. >>> >>>-Dror >>> >>>------------------------------------------------------------------------ >>> >>>_______________________________________________ >>>IPoverIB mailing list >>>IPoverIB@ietf.org >>>https://www1.ietf.org/mailman/listinfo/ipoverib >>> >> >> >> >>_______________________________________________ >>IPoverIB mailing list >>IPoverIB@ietf.org >>https://www1.ietf.org/mailman/listinfo/ipoverib > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Tue Oct 25 14:18:15 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUTNH-0003lq-4f; Tue, 25 Oct 2005 14:18:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUTND-0003jj-2y for ipoverib@megatron.ietf.org; Tue, 25 Oct 2005 14:18:13 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10276 for ; Tue, 25 Oct 2005 14:17:56 -0400 (EDT) Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUTaD-0004zf-8o for ipoverib@ietf.org; Tue, 25 Oct 2005 14:31:37 -0400 Received: from tsx2.cup.hp.com (tsx2.cup.hp.com [15.13.185.29]) by palrel10.hp.com (Postfix) with ESMTP id 50CF01B31; Tue, 25 Oct 2005 11:18:09 -0700 (PDT) Received: from expvr082905.hp.com ([15.244.201.166]) by tsx2.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id LAA00171; Tue, 25 Oct 2005 11:18:08 -0700 (PDT) Message-Id: <6.2.3.4.0.20051025111649.02f74eb0@tsx2.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Tue, 25 Oct 2005 11:18:05 -0700 To: "H.K. Jerry Chu" , krause@cup.hp.com, bill@strahm.net From: Vandana Rao Subject: Re: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <200510241741.j9OHfpHu274157@jurassic.eng.sun.com> References: <200510241741.j9OHfpHu274157@jurassic.eng.sun.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: e1b0e72ff1bbd457ceef31828f216a86 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Yes, Jerry is right. This does not have to do with how TCP connections are mapped to IB connections but whether there is an issue with multiple active-active IB connections between 2 peers. I believe there is none. -Vandana At 10:40 AM 10/24/2005, H.K. Jerry Chu wrote: >I think Dror and Vandana are talking about connection "establishment", >not how to use multiple connections after they have been established. >It looks like there are some robustness issues wrt the handshake for >setting up multiple IB connections. > >But I may be wrong of course. > >Jerry > > >At 08:32 PM 10/21/2005, Bill Strahm wrote: > >>Let me try to understand what you are worried about: > >>I am assuming that each TCP connection (5-tuple) can be on its own CM > >>connection, and not multiplex a single TCP connection over multiple CM > >>connections. > >> > >>Is this your understanding ? > >> > >>Now if this is what we are talking about - we don't have a problem. TCP > >>only guarantees in order delivery to a single connection - if I have > >>multiple TCP connections to the same hosts, IP packets can be completely > >>reordered/jumbled by the intermediate network - and all I am guaranteed > >>is that each TCP stream will have in order delivery relative to itself. > >> > >>I am not aware of any applications where in order IP packet delivery is > >>required between multiple streams. If this is the case - it has to be > >>handled above TCP anyway, as TCP does not make this guarantee. > > > >>Comments ? > > > >Your understanding is correct. IP packets can arrive in any order on any > >interface though all implementations prefer them to arrive close to in > >order and on the same interface (Ethernet aggregation technology attempts > >to maintain the same interface is used consistently). Now, having multiple > >IB connections between endnode pairs is not an issue - does not matter what > >VL, path, etc. is taken. However, ideally, the IP datagrams for a given > >connection are injected for a given tuple on the same path consistently as > >this leads to optimal performance - it is not a correctness issue though as > >ordering is the TCP layer's responsibility. > > > >Mike > > > >>Bill > >> > >>Dror Goldenberg wrote: > >> > >>>I think that the current model for multiple connections > establishment between > >>>two peers works well for 1 connection but is broken for more than one > >>>connection. > >>>Because of cases where CM packets are dropped or travel on different VLs, > >>>they > >>>might get reordered on the way and observed in a different order than > >>>were originally > >>>sent. This breaks the current model which assumes that the REQs > are sent and > >>>processed in order. > >>> > >>>I think that we need to think of a model which is robust to CM packets > >>>reordering > >>>which can be one of: > >>>- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. > >>>It helps > >>> synchronizing both peers on which is the channel at question now. The > >>> rest > >>> of the decision (per channel) of whether to accept or reject can be > >>> the same > >>> as defined in the draft. > >>>- Not supporting multiple connections between the same peers. > >>> > >>>-Dror > >>> > >>>------------------------------------------------------------------------ > >>> > >>>_______________________________________________ > >>>IPoverIB mailing list > >>>IPoverIB@ietf.org > >>>https://www1.ietf.org/mailman/listinfo/ipoverib > >>> > >> > >> > >> > >>_______________________________________________ > >>IPoverIB mailing list > >>IPoverIB@ietf.org > >>https://www1.ietf.org/mailman/listinfo/ipoverib > > > > >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Tue Oct 25 17:48:37 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUWer-0004GF-Nz; Tue, 25 Oct 2005 17:48:37 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUWep-0004FZ-V9 for ipoverib@megatron.ietf.org; Tue, 25 Oct 2005 17:48:36 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA16855 for ; Tue, 25 Oct 2005 17:48:20 -0400 (EDT) Received: from e33.co.us.ibm.com ([32.97.110.151]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUWrs-0003H3-3F for ipoverib@ietf.org; Tue, 25 Oct 2005 18:02:04 -0400 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by e33.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9PLmMLk031285 for ; Tue, 25 Oct 2005 17:48:22 -0400 Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d03relay04.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9PLnK53527268 for ; Tue, 25 Oct 2005 15:49:20 -0600 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9PLmLMm028875 for ; Tue, 25 Oct 2005 15:48:21 -0600 Received: from dyn9047018063.beaverton.ibm.com (dyn9047018063.beaverton.ibm.com [9.47.18.63]) by d03av01.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9PLmLTq028846; Tue, 25 Oct 2005 15:48:21 -0600 Date: Tue, 25 Oct 2005 14:37:38 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: Vandana Rao Subject: RE: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6.2.3.4.0.20051020102136.02dfe338@tsx2.cup.hp.com> Message-ID: References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA5A@mtlexch01.mtl.com> <6.2.3.4.0.20051020102136.02dfe338@tsx2.cup.hp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: 2857c5c041d6c02d7181d602c22822c8 Cc: Dror Goldenberg , "'ipoverib@ietf.org'" X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Hi Vandana, I too agree there is no problem here. I think you are right if the left peer accepts the connection. However, if the reverse was the case, that is the left peer rejected the connection since it is found that the local address is numerically larger. It will do this comparison since its REQ is also outstanding. The right peer will not receive any REQ (it is delayed or yet to be retransmitted). If it receives the reject it will cancel its request. At a later time it will receive the delayed/retransmitted request and one connection will be established. Dror, do these two scenarios look right to you? Vivek On Thu, 20 Oct 2005, Vandana Rao wrote: > Hi Dror, > I think you are confusing the 2 IDs that come into play in this setup. The > communication ID is something that the CM entity on a node understands but > the transaction ID is something that the MAD handling entity uses (for > retransmission, req/resp matchup etc.). > So, in the second slide, there should not be a retransmission of the left > peer REQ MAD since when the left peer accepts the connection, it will cancel > its outstanding REQ request and hence the retransmission should not take > place. > -Vandana > > At 12:39 AM 10/20/2005, Dror Goldenberg wrote: >> After giving it some more thought, I think that there is still a problem. >> Please look at the attached pdf. >> >> In the first slide, everything happens smoothly, the left peer knows that >> it has to accept the connection and the right peer knows that it has to >> decline. >> On the 2nd slide, the left peer REQ transmission is dropped in the fabric. >> It is retransmitted independently by the CM way after a channel has already >> been established. Now the right side can not really tell whether this is a >> new channel request or an old "simultaneous connection channel". This may >> lead to two channels being opened mistakenly. >> >> I don't believe it can be solved by looking at the communication ID. This >> number is just being fabricated by the sender of the REQ and is just a >> "random number" with respect to what we're trying to determine. >> >> -Dror >> >> -----Original Message----- >> From: Dror Goldenberg >> Sent: Wednesday, October 19, 2005 9:37 PM >> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >> >> Hi Vandana, >> >> Thanks for clarifying this. I agree that if IPoiB-CM implementation looks >> at the Communication ID it can make those decisions correctly. The question >> is whether this value is exposed through common CM APIs. For all other >> purposes, the CM doesn't need to expose the Communication ID to the ULP. >> However, seems like it's not a big deal to expose this value to the ULP. >> Anyway, it worth noting this explanation in the RFC so that it's clear that >> a proper implementation of IPoIB-CM should monitor Communication ID as part >> of the simultaneous connection decision. >> >> -Dror >> -----Original Message----- >> From: Vandana Rao [mailto:vandana.rao@hp.com] >> Sent: Wednesday, October 19, 2005 6:40 PM >> To: Dror Goldenberg; ipoverib@ietf.org >> Subject: Re: [Ipoverib] ipoib-cm multiple connection model >> >> Hi Dror, >> The purpose of the MAD transaction ID is to allow one to do what you say >> below, i.e. have multiple connection establishment MADs between 2 peers. It >> is essentially the same as the "channel ID" that you talk about below. Its >> just that it is a generic channel ID for any MAD transactions, not just CM >> MADs. >> I do not believe there is any issue with existing IB CM in this regard. >> -Vandana >> >> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >>> I think that the current model for multiple connections establishment >>> between >>> two peers works well for 1 connection but is broken for more than one >>> connection. >>> Because of cases where CM packets are dropped or travel on different VLs, >>> they >>> might get reordered on the way and observed in a different order than were >>> originally >>> sent. This breaks the current model which assumes that the REQs are sent >>> and >>> processed in order. >>> >>> I think that we need to think of a model which is robust to CM packets >>> reordering >>> which can be one of: >>> - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. >>> It helps >>> synchronizing both peers on which is the channel at question now. The >>> rest >>> of the decision (per channel) of whether to accept or reject can be the >>> same >>> as defined in the draft. >>> - Not supporting multiple connections between the same peers. >>> >>> -Dror >>> _______________________________________________ >>> IPoverIB mailing list >>> IPoverIB@ietf.org >>> https://www1.ietf.org/mailman/listinfo/ipoverib >> > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Wed Oct 26 03:17:13 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfX7-0007Qc-Oo; Wed, 26 Oct 2005 03:17:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfX5-0007NI-CN for ipoverib@megatron.ietf.org; Wed, 26 Oct 2005 03:17:11 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA28403 for ; Wed, 26 Oct 2005 03:16:56 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUfk9-0006cu-IX for ipoverib@ietf.org; Wed, 26 Oct 2005 03:30:44 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Wed, 26 Oct 2005 09:16:07 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9C@mtlexch01.mtl.com> From: Dror Goldenberg To: Bill Strahm Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Wed, 26 Oct 2005 09:20:55 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.5 (/) X-Scan-Signature: 7698d1420ecbbce1995432e99bb6d1a1 Cc: ipoverib@ietf.org X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1745897787==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============1745897787== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D9FD.CE2A2930" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D9FD.CE2A2930 Content-Type: text/plain As Jerry and Vandana mentioned I was referring to the IPoIB-CM connection establishment and not to how IP datagrams are striped amongst them. As of the point you're raising, maybe it worth noting it in the RFC draft. E.g. when multiple connections are being used between two IPoIB-CM peers, for optimal performance it is desirable to send IP datagrams of the same TCP/UDP session on the same IPoIB-CM connection. A common implementation method is to choose the connection according to a hash value calculated on the TCP' 5-tuple. -Dror > -----Original Message----- > From: Bill Strahm [mailto:bill@strahm.net] > Sent: Saturday, October 22, 2005 5:33 AM > To: Dror Goldenberg > Cc: ipoverib@ietf.org > Subject: Re: [Ipoverib] ipoib-cm multiple connection model > > > Let me try to understand what you are worried about: > I am assuming that each TCP connection (5-tuple) can be on > its own CM connection, and not multiplex a single TCP > connection over multiple CM connections. > > Is this your understanding ? > > Now if this is what we are talking about - we don't have a > problem. TCP only guarantees in order delivery to a single > connection - if I have multiple TCP connections to the same > hosts, IP packets can be completely reordered/jumbled by the > intermediate network - and all I am guaranteed is that each > TCP stream will have in order delivery relative to itself. > > I am not aware of any applications where in order IP packet > delivery is required between multiple streams. If this is > the case - it has to be handled above TCP anyway, as TCP does > not make this guarantee. > > Comments ? > Bill > > Dror Goldenberg wrote: > > > I think that the current model for multiple connections > establishment > > between > > two peers works well for 1 connection but is broken for > more than one > > connection. > > Because of cases where CM packets are dropped or travel on > different > > VLs, they > > might get reordered on the way and observed in a different > order than > > were originally > > sent. This breaks the current model which assumes that the REQs are > > sent and > > processed in order. > > > > I think that we need to think of a model which is robust to > CM packets > > reordering > > which can be one of: > > - Adding a "channel ID" in the private data. The field goes > 0,1,2 and > > on. It helps > > synchronizing both peers on which is the channel at > question now. > > The rest > > of the decision (per channel) of whether to accept or > reject can be > > the same > > as defined in the draft. > > - Not supporting multiple connections between the same peers. > > > > -Dror > > > >------------------------------------------------------------- > ---------- > >- > > > >_______________________________________________ > >IPoverIB mailing list > >IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib > > > > > ------_=_NextPart_001_01C5D9FD.CE2A2930 Content-Type: text/html Content-Transfer-Encoding: quoted-printable RE: [Ipoverib] ipoib-cm multiple connection model

As Jerry and Vandana  mentioned I was referring = to the IPoIB-CM connection establishment and not to how IP datagrams = are striped amongst them.

As of the point you're raising, maybe it worth noting = it in the RFC draft. E.g. when multiple connections are being used = between two IPoIB-CM peers, for optimal performance it is desirable to = send IP datagrams of the same TCP/UDP session on the same IPoIB-CM = connection. A common implementation method is to choose the connection = according to a hash value calculated on the TCP' 5-tuple.

-Dror

> -----Original Message-----
> From: Bill Strahm [mailto:bill@strahm.net]
> Sent: Saturday, October 22, 2005 5:33 AM
> To: Dror Goldenberg
> Cc: ipoverib@ietf.org
> Subject: Re: [Ipoverib] ipoib-cm multiple = connection model
>
>
> Let me try to understand what you are worried = about:
> I am assuming that each TCP connection = (5-tuple) can be on
> its own CM connection, and not multiplex a = single TCP
> connection over multiple CM connections.
>
> Is this your understanding ?
>
> Now if this is what we are talking about - we = don't have a
> problem.  TCP only guarantees in order = delivery to a single
> connection - if I have multiple TCP connections = to the same
> hosts, IP packets can be completely = reordered/jumbled by the
> intermediate network - and all I am guaranteed = is that each
> TCP stream will have in order delivery relative = to itself.
>
> I am not aware of any applications where in = order IP packet
> delivery is required between multiple = streams.  If this is
> the case - it has to be handled above TCP = anyway, as TCP does
> not make this guarantee.
>
> Comments ?
> Bill
>
> Dror Goldenberg wrote:
>
> > I think that the current model for = multiple connections
> establishment
> > between
> > two peers works well for 1 connection but = is broken for
> more than one
> > connection.
> > Because of cases where CM packets are = dropped or travel on
> different
> > VLs, they
> > might get reordered on the way and = observed in a different
> order than
> > were originally
> > sent. This breaks the current model which = assumes that the REQs are
> > sent and
> > processed in order.
> > 
> > I think that we need to think of a model = which is robust to
> CM packets
> > reordering
> > which can be one of:
> > - Adding a "channel ID" in the = private data. The field goes
> 0,1,2 and
> > on. It helps
> >    synchronizing both peers = on which is the channel at
> question now.
> > The rest
> >    of the decision (per = channel) of whether to accept or
> reject can be
> > the same
> >    as defined in the = draft.
> > - Not supporting multiple connections = between the same peers.
> > 
> > -Dror
> >
> = >-------------------------------------------------------------=
> ----------
> >-
> >
> = >_______________________________________________
> >IPoverIB mailing list
> >IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib
> > 
> >
>

------_=_NextPart_001_01C5D9FD.CE2A2930-- --===============1745897787== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1745897787==-- From ipoverib-bounces@ietf.org Wed Oct 26 03:17:14 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfX8-0007RG-2w; Wed, 26 Oct 2005 03:17:14 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfX5-0007Ne-Ff for ipoverib@megatron.ietf.org; Wed, 26 Oct 2005 03:17:11 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA28406 for ; Wed, 26 Oct 2005 03:16:56 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUfk9-0006d3-IY for ipoverib@ietf.org; Wed, 26 Oct 2005 03:30:44 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Wed, 26 Oct 2005 09:16:10 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9D@mtlexch01.mtl.com> From: Dror Goldenberg To: Vandana Rao , Dror Goldenberg , Dror Goldenberg , "'ipoverib@ietf.org'" Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Wed, 26 Oct 2005 09:20:57 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.1 (/) X-Scan-Signature: 17cf8eab1d6bbd2874a56f9e3554d91d Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============2079530104==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============2079530104== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D9FD.CEE682F6" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D9FD.CEE682F6 Content-Type: text/plain Hi Vandana, The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases: 1) The CM uses more than one QP, in which case the HCA can reorder those packets 2) The CM uses different SL/VL, in which case the fabric can reorder the packets 3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU. My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required. -Dror -----Original Message----- From: Vandana Rao [mailto:vandana.rao@hp.com] Sent: Thursday, October 20, 2005 7:25 PM To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org' Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Dror, I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.). So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place. -Vandana At 12:39 AM 10/20/2005, Dror Goldenberg wrote: After giving it some more thought, I think that there is still a problem. Please look at the attached pdf. In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline. On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly. I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine. -Dror -----Original Message----- From: Dror Goldenberg Sent: Wednesday, October 19, 2005 9:37 PM To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Vandana, Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision. -Dror -----Original Message----- From: Vandana Rao [ mailto:vandana.rao@hp.com ] Sent: Wednesday, October 19, 2005 6:40 PM To: Dror Goldenberg; ipoverib@ietf.org Subject: Re: [Ipoverib] ipoib-cm multiple connection model Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib ------_=_NextPart_001_01C5D9FD.CEE682F6 Content-Type: text/html Message
Hi Vandana,
 
The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases:
1) The CM uses more than one QP, in which case the HCA can reorder those packets
2) The CM uses different SL/VL, in which case the fabric can reorder the packets
3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU.
 
My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required.
 
-Dror
-----Original Message-----
From: Vandana Rao [mailto:vandana.rao@hp.com]
Sent: Thursday, October 20, 2005 7:25 PM
To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org'
Subject: RE: [Ipoverib] ipoib-cm multiple connection model

Hi Dror,
I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.).
 So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place.
-Vandana

At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.
 
In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.
 
I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.
 
-Dror
 
-----Original Message-----
From: Dror Goldenberg
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Vandana,

 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.

 
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana

At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.

 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.

 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib

------_=_NextPart_001_01C5D9FD.CEE682F6-- --===============2079530104== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============2079530104==-- From ipoverib-bounces@ietf.org Wed Oct 26 03:17:17 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfXB-0007Ue-M2; Wed, 26 Oct 2005 03:17:17 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUfX6-0007OV-33 for ipoverib@megatron.ietf.org; Wed, 26 Oct 2005 03:17:15 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA28407 for ; Wed, 26 Oct 2005 03:16:56 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUfk9-0006cn-IX for ipoverib@ietf.org; Wed, 26 Oct 2005 03:30:44 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Wed, 26 Oct 2005 09:16:03 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9A@mtlexch01.mtl.com> From: Dror Goldenberg To: Vivek Kashyap , Vandana Rao Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Wed, 26 Oct 2005 09:18:00 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.8 (/) X-Scan-Signature: 031b5f42d6d2cd097710e6b68613cb36 Cc: Dror Goldenberg , "'ipoverib@ietf.org'" X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0540686602==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0540686602== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5D9FD.CD85A6EE" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5D9FD.CD85A6EE Content-Type: text/plain Vivek, I agree that the two scenarios you mention here are the interesting ones. I don't agree about the solution. Let's look at the two cases: 1) Left peer accepts I disagree. You have an assumption that left peer can revoke its original connection request - BEFORE transmitting the REP that accepts the peer request. I am not sure that you can do it with every CM implementation. Even if you can, then the HCA can reorder those packets if they are on different QPs, and the fabric can reorder them as well if they are on different VLs. 2) Left peer rejects I think that this scenario works. Thanks Dror > -----Original Message----- > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] > Sent: Tuesday, October 25, 2005 11:38 PM > To: Vandana Rao > Cc: Dror Goldenberg; 'ipoverib@ietf.org' > Subject: RE: [Ipoverib] ipoib-cm multiple connection model > > > Hi Vandana, > > I too agree there is no problem here. > > I think you are right if the left peer accepts the > connection. However, if the reverse was the case, that is the > left peer rejected the connection since it is found that the > local address is numerically larger. It will do this > comparison since its REQ is also outstanding. The right peer > will not receive any REQ (it is delayed or yet to be > retransmitted). If it receives the reject it will cancel its > request. At a later time it will receive the > delayed/retransmitted request and one connection will be > established. > > Dror, do these two scenarios look right to you? > > Vivek > > On Thu, 20 Oct 2005, Vandana Rao wrote: > > > Hi Dror, > > I think you are confusing the 2 IDs that come into play in > this setup. > > The > > communication ID is something that the CM entity on a node > understands but > > the transaction ID is something that the MAD handling > entity uses (for > > retransmission, req/resp matchup etc.). > > So, in the second slide, there should not be a > retransmission of the left > > peer REQ MAD since when the left peer accepts the > connection, it will cancel > > its outstanding REQ request and hence the retransmission > should not take > > place. > > -Vandana > > > > At 12:39 AM 10/20/2005, Dror Goldenberg wrote: > >> After giving it some more thought, I think that there is still a > >> problem. > >> Please look at the attached pdf. > >> > >> In the first slide, everything happens smoothly, the left > peer knows > >> that > >> it has to accept the connection and the right peer knows > that it has to > >> decline. > >> On the 2nd slide, the left peer REQ transmission is > dropped in the fabric. > >> It is retransmitted independently by the CM way after a > channel has already > >> been established. Now the right side can not really tell > whether this is a > >> new channel request or an old "simultaneous connection > channel". This may > >> lead to two channels being opened mistakenly. > >> > >> I don't believe it can be solved by looking at the > communication ID. > >> This > >> number is just being fabricated by the sender of the REQ > and is just a > >> "random number" with respect to what we're trying to determine. > >> > >> -Dror > >> > >> -----Original Message----- > >> From: Dror Goldenberg > >> Sent: Wednesday, October 19, 2005 9:37 PM > >> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org > >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >> > >> Hi Vandana, > >> > >> Thanks for clarifying this. I agree that if IPoiB-CM > implementation > >> looks > >> at the Communication ID it can make those decisions > correctly. The question > >> is whether this value is exposed through common CM APIs. > For all other > >> purposes, the CM doesn't need to expose the Communication > ID to the ULP. > >> However, seems like it's not a big deal to expose this > value to the ULP. > >> Anyway, it worth noting this explanation in the RFC so > that it's clear that > >> a proper implementation of IPoIB-CM should monitor > Communication ID as part > >> of the simultaneous connection decision. > >> > >> -Dror > >> -----Original Message----- > >> From: Vandana Rao [mailto:vandana.rao@hp.com] > >> Sent: Wednesday, October 19, 2005 6:40 PM > >> To: Dror Goldenberg; ipoverib@ietf.org > >> Subject: Re: [Ipoverib] ipoib-cm multiple connection model > >> > >> Hi Dror, > >> The purpose of the MAD transaction ID is to allow one to > do what you > >> say > >> below, i.e. have multiple connection establishment MADs > between 2 peers. It > >> is essentially the same as the "channel ID" that you talk > about below. Its > >> just that it is a generic channel ID for any MAD > transactions, not just CM > >> MADs. > >> I do not believe there is any issue with existing IB CM in > this regard. > >> -Vandana > >> > >> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: > >>> I think that the current model for multiple connections > >>> establishment > >>> between > >>> two peers works well for 1 connection but is broken for > more than one > >>> connection. > >>> Because of cases where CM packets are dropped or travel > on different VLs, > >>> they > >>> might get reordered on the way and observed in a > different order than were > >>> originally > >>> sent. This breaks the current model which assumes that > the REQs are sent > >>> and > >>> processed in order. > >>> > >>> I think that we need to think of a model which is robust to CM > >>> packets > >>> reordering > >>> which can be one of: > >>> - Adding a "channel ID" in the private data. The field > goes 0,1,2 and on. > >>> It helps > >>> synchronizing both peers on which is the channel at > question now. The > >>> rest > >>> of the decision (per channel) of whether to accept or > reject can be the > >>> same > >>> as defined in the draft. > >>> - Not supporting multiple connections between the same peers. > >>> > >>> -Dror > >>> _______________________________________________ > >>> IPoverIB mailing list > >>> IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib > >> > > > ------_=_NextPart_001_01C5D9FD.CD85A6EE Content-Type: text/html RE: [Ipoverib] ipoib-cm multiple connection model

Vivek,

I agree that the two scenarios you mention here are the interesting ones.
I don't agree about the solution. Let's look at the two cases:

1) Left peer accepts
        I disagree. You have an assumption that left peer can
        revoke its original connection request - BEFORE transmitting
        the REP that accepts the peer request. I am not sure that
        you can do it with every CM implementation. Even if you can,
        then the HCA can reorder those packets if they are on
        different QPs, and the fabric can reorder them as well if
        they are on different VLs.

2) Left peer rejects
        I think that this scenario works.

Thanks
Dror

> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Tuesday, October 25, 2005 11:38 PM
> To: Vandana Rao
> Cc: Dror Goldenberg; 'ipoverib@ietf.org'
> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>
>
> Hi Vandana,
>
> I too agree there is no problem here.
>
> I think you are right if the left peer accepts the
> connection. However, if the reverse was the case, that is the
> left peer rejected the connection since it is found that the
> local address is numerically larger. It will do this
> comparison since its REQ is also outstanding.  The right peer
> will not receive any REQ  (it is delayed or yet to be
> retransmitted). If it receives the reject it will cancel its
> request. At a later time it will receive the
> delayed/retransmitted request and one connection will be
> established.
>
> Dror, do these two scenarios look right to you?
>
> Vivek
>
> On Thu, 20 Oct 2005, Vandana Rao wrote:
>
> > Hi Dror,
> > I think you are confusing the 2 IDs that come into play in
> this setup.
> > The
> > communication ID is something that the CM entity on a node
> understands but
> > the transaction ID is something that the MAD handling
> entity uses (for
> > retransmission, req/resp matchup etc.).
> > So, in the second slide, there should not be a
> retransmission of the left
> > peer REQ MAD since when the left peer accepts the
> connection, it will cancel
> > its outstanding REQ request and hence the retransmission
> should not take
> > place.
> > -Vandana
> >
> > At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
> >> After giving it some more thought, I think that there is still a
> >> problem.
> >> Please look at the attached pdf.
> >>
> >> In the first slide, everything happens smoothly, the left
> peer knows
> >> that
> >> it has to accept the connection and the right peer knows
> that it has to
> >> decline.
> >> On the 2nd slide, the left peer REQ transmission is
> dropped in the fabric.
> >> It is retransmitted independently by the CM way after a
> channel has already
> >> been established. Now the right side can not really tell
> whether this is a
> >> new channel request or an old "simultaneous connection
> channel". This may
> >> lead to two channels being opened mistakenly.
> >>
> >> I don't believe it can be solved by looking at the
> communication ID.
> >> This
> >> number is just being fabricated by the sender of the REQ
> and is just a
> >> "random number" with respect to what we're trying to determine.
> >>
> >> -Dror
> >>
> >> -----Original Message-----
> >> From: Dror Goldenberg
> >> Sent: Wednesday, October 19, 2005 9:37 PM
> >> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
> >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>
> >> Hi Vandana,
> >>
> >> Thanks for clarifying this. I agree that if IPoiB-CM
> implementation
> >> looks
> >> at the Communication ID it can make those decisions
> correctly. The question
> >> is whether this value is exposed through common CM APIs.
> For all other
> >> purposes, the CM doesn't need to expose the Communication
> ID to the ULP.
> >> However, seems like it's not a big deal to expose this
> value to the ULP.
> >> Anyway, it worth noting this explanation in the RFC so
> that it's clear that
> >> a proper implementation of IPoIB-CM should monitor
> Communication ID as part
> >> of the simultaneous connection decision.
> >>
> >> -Dror
> >> -----Original Message-----
> >> From: Vandana Rao [mailto:vandana.rao@hp.com]
> >> Sent: Wednesday, October 19, 2005 6:40 PM
> >> To: Dror Goldenberg; ipoverib@ietf.org
> >> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
> >>
> >> Hi Dror,
> >> The purpose of the MAD transaction ID is to allow one to
> do what you
> >> say
> >> below, i.e. have multiple connection establishment MADs
> between 2 peers. It
> >> is essentially the same as the "channel ID" that you talk
> about below. Its
> >> just that it is a generic channel ID for any MAD
> transactions, not just CM
> >> MADs.
> >> I do not believe there is any issue with existing IB CM in
> this regard.
> >> -Vandana
> >>
> >> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
> >>> I think that the current model for multiple connections
> >>> establishment
> >>> between
> >>> two peers works well for 1 connection but is broken for
> more than one
> >>> connection.
> >>> Because of cases where CM packets are dropped or travel
> on different VLs,
> >>> they
> >>> might get reordered on the way and observed in a
> different order than were
> >>> originally
> >>> sent. This breaks the current model which assumes that
> the REQs are sent
> >>> and
> >>> processed in order.
> >>>
> >>> I think that we need to think of a model which is robust to CM
> >>> packets
> >>> reordering
> >>> which can be one of:
> >>> - Adding a "channel ID" in the private data. The field
> goes 0,1,2 and on.
> >>> It helps
> >>>    synchronizing both peers on which is the channel at
> question now. The
> >>> rest
> >>>    of the decision (per channel) of whether to accept or
> reject can be the
> >>> same
> >>>    as defined in the draft.
> >>> - Not supporting multiple connections between the same peers.
> >>>
> >>> -Dror
> >>> _______________________________________________
> >>> IPoverIB mailing list
> >>> IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib
> >>
> >
>

------_=_NextPart_001_01C5D9FD.CD85A6EE-- --===============0540686602== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0540686602==-- From ipoverib-bounces@ietf.org Wed Oct 26 14:36:35 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUq8Y-0006CH-Kl; Wed, 26 Oct 2005 14:36:34 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUq8T-00069b-Lh for ipoverib@megatron.ietf.org; Wed, 26 Oct 2005 14:36:32 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA13180 for ; Wed, 26 Oct 2005 14:36:14 -0400 (EDT) Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUqLg-0004wI-49 for ipoverib@ietf.org; Wed, 26 Oct 2005 14:50:08 -0400 Received: from tsx2.cup.hp.com (tsx2.cup.hp.com [15.13.185.29]) by palrel10.hp.com (Postfix) with ESMTP id 8069B298A; Wed, 26 Oct 2005 11:34:54 -0700 (PDT) Received: from expvr082905.hp.com (vr724251.cup.hp.com [15.13.108.169]) by tsx2.cup.hp.com (8.9.3 (PHNE_29774)/8.8.6) with ESMTP id LAA12364; Wed, 26 Oct 2005 11:34:51 -0700 (PDT) Message-Id: <6.2.3.4.0.20051026112138.02cebe08@tsx2.cup.hp.com> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Wed, 26 Oct 2005 11:34:50 -0700 To: Dror Goldenberg , "'ipoverib@ietf.org'" From: Vandana Rao Subject: RE: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9D@mtlexch01.mtl.com> References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9D@mtlexch01.mtl.com> Mime-Version: 1.0 X-Spam-Score: 0.1 (/) X-Scan-Signature: fcb459c204557d9509ce9c1b55d771f1 Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============1220309086==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org --===============1220309086== Content-Type: multipart/alternative; boundary="=====================_701526201==.ALT" --=====================_701526201==.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Hi Dror, You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP. If not, there is potential for the state machines to get messed up. -Vandana At 12:20 AM 10/26/2005, Dror Goldenberg wrote: >Hi Vandana, > >The ability to revoke a request is a CM API issue that may or may >not be present in an implementation. Assuming it is present, you >might be late to revoke, e.g. the CM has already scheduled the REQ >to be retransmitted. In which case, you have a very strong >assumption that the retransmitted REQ will go out to the wire BEFORE >the REP will. This assumption may break in the following cases: >1) The CM uses more than one QP, in which case the HCA can reorder >those packets >2) The CM uses different SL/VL, in which case the fabric can reorder >the packets >3) The CM may choose to internally queue the packets and then submit >them to the HCA. This internal queuing mechanism can reorder >packets, e.g. when you have a queue per CPU. > >My feeling is that relying on packet ordering between different >unrelated streams won't be robust, and can cause more connections to >be established than were actually required. > >-Dror >-----Original Message----- >From: Vandana Rao [mailto:vandana.rao@hp.com] >Sent: Thursday, October 20, 2005 7:25 PM >To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org' >Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >Hi Dror, >I think you are confusing the 2 IDs that come into play in this >setup. The communication ID is something that the CM entity on a >node understands but the transaction ID is something that the MAD >handling entity uses (for retransmission, req/resp matchup etc.). > So, in the second slide, there should not be a retransmission of > the left peer REQ MAD since when the left peer accepts the > connection, it will cancel its outstanding REQ request and hence > the retransmission should not take place. >-Vandana > >At 12:39 AM 10/20/2005, Dror Goldenberg wrote: >>After giving it some more thought, I think that there is still a >>problem. Please look at the attached pdf. >> >>In the first slide, everything happens smoothly, the left peer >>knows that it has to accept the connection and the right peer knows >>that it has to decline. >>On the 2nd slide, the left peer REQ transmission is dropped in the >>fabric. It is retransmitted independently by the CM way after a >>channel has already been established. Now the right side can not >>really tell whether this is a new channel request or an old >>"simultaneous connection channel". This may lead to two channels >>being opened mistakenly. >> >>I don't believe it can be solved by looking at the communication >>ID. This number is just being fabricated by the sender of the REQ >>and is just a "random number" with respect to what we're trying to determine. >> >>-Dror >> >>-----Original Message----- >>From: Dror Goldenberg >>Sent: Wednesday, October 19, 2005 9:37 PM >>To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org >>Subject: RE: [Ipoverib] ipoib-cm multiple connection model >> >>Hi Vandana, >> >>Thanks for clarifying this. I agree that if IPoiB-CM implementation >>looks at the Communication ID it can make those decisions >>correctly. The question is whether this value is exposed through >>common CM APIs. For all other purposes, the CM doesn't need to >>expose the Communication ID to the ULP. However, seems like it's >>not a big deal to expose this value to the ULP. >>Anyway, it worth noting this explanation in the RFC so that it's >>clear that a proper implementation of IPoIB-CM should monitor >>Communication ID as part of the simultaneous connection decision. >> >>-Dror >>-----Original Message----- >>From: Vandana Rao [ mailto:vandana.rao@hp.com] >>Sent: Wednesday, October 19, 2005 6:40 PM >>To: Dror Goldenberg; ipoverib@ietf.org >>Subject: Re: [Ipoverib] ipoib-cm multiple connection model >> >>Hi Dror, >>The purpose of the MAD transaction ID is to allow one to do what >>you say below, i.e. have multiple connection establishment MADs >>between 2 peers. It is essentially the same as the "channel ID" >>that you talk about below. Its just that it is a generic channel ID >>for any MAD transactions, not just CM MADs. >>I do not believe there is any issue with existing IB CM in this regard. >>-Vandana >>At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >>>I think that the current model for multiple connections >>>establishment between >>>two peers works well for 1 connection but is broken for more than >>>one connection. >>>Because of cases where CM packets are dropped or travel on >>>different VLs, they >>>might get reordered on the way and observed in a different order >>>than were originally >>>sent. This breaks the current model which assumes that the REQs >>>are sent and >>>processed in order. >>> >>>I think that we need to think of a model which is robust to CM >>>packets reordering >>>which can be one of: >>>- Adding a "channel ID" in the private data. The field goes 0,1,2 >>>and on. It helps >>> synchronizing both peers on which is the channel at question >>> now. The rest >>> of the decision (per channel) of whether to accept or reject >>> can be the same >>> as defined in the draft. >>>- Not supporting multiple connections between the same peers. >>> >>>-Dror >>>_______________________________________________ >>>IPoverIB mailing list >>>IPoverIB@ietf.org >>>https://www1.ietf.org/mailman/listinfo/ipoverib >_______________________________________________ >IPoverIB mailing list >IPoverIB@ietf.org >https://www1.ietf.org/mailman/listinfo/ipoverib --=====================_701526201==.ALT Content-Type: text/html; charset="us-ascii" Hi Dror,
You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP.
If not, there is potential for the state machines to get messed up.
-Vandana

At 12:20 AM 10/26/2005, Dror Goldenberg wrote:
Hi Vandana,
 
The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases:
1) The CM uses more than one QP, in which case the HCA can reorder those packets
2) The CM uses different SL/VL, in which case the fabric can reorder the packets
3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU.
 
My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required.
 
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Thursday, October 20, 2005 7:25 PM
To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org'
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.).
 So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place.
-Vandana

At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.
 
In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.
 
I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.
 
-Dror
 
-----Original Message-----
From: Dror Goldenberg
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Vandana,
 
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.
 
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana
At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
 
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
 
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
--=====================_701526201==.ALT-- --===============1220309086== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============1220309086==-- From ipoverib-bounces@ietf.org Wed Oct 26 16:44:31 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUs8N-0001Mv-3r; Wed, 26 Oct 2005 16:44:31 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EUs8K-0001Lu-Ou for ipoverib@megatron.ietf.org; Wed, 26 Oct 2005 16:44:29 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13101 for ; Wed, 26 Oct 2005 16:44:12 -0400 (EDT) Received: from e35.co.us.ibm.com ([32.97.110.153]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EUsLH-0008Vi-G1 for ipoverib@ietf.org; Wed, 26 Oct 2005 16:58:08 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e35.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9QKhsAf004090 for ; Wed, 26 Oct 2005 16:43:54 -0400 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9QKhstF516312 for ; Wed, 26 Oct 2005 14:43:54 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9QKhssA002636 for ; Wed, 26 Oct 2005 14:43:54 -0600 Received: from dyn9047018063.beaverton.ibm.com (dyn9047018063.beaverton.ibm.com [9.47.18.63]) by d03av03.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9QKhrcb002551; Wed, 26 Oct 2005 14:43:54 -0600 Date: Wed, 26 Oct 2005 13:33:02 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: Dror Goldenberg Subject: RE: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9A@mtlexch01.mtl.com> Message-ID: References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CA9A@mtlexch01.mtl.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: a3f7094ccc62748c06b21fcf44c073ee Cc: "'ipoverib@ietf.org'" , Vandana Rao X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org On Wed, 26 Oct 2005, Dror Goldenberg wrote: > Vivek, > > I agree that the two scenarios you mention here are the interesting ones. > I don't agree about the solution. Let's look at the two cases: > > 1) Left peer accepts > I disagree. You have an assumption that left peer can > revoke its original connection request - BEFORE transmitting > the REP that accepts the peer request. I am not sure that > you can do it with every CM implementation. Even if you can, > then the HCA can reorder those packets if they are on > different QPs, and the fabric can reorder them as well if > they are on different VLs. Yes, that has the potential for extra connections..however, the following will occur if CM checks don't work. Left peer accepted. Sent a REP. And of course the REQ was also sent. The right receiver can receive : a. REP first It sets up the connection. Then receives a REQ. i) If it accepts multiple connections from the same peer: it will REP with a new QP (connection). Now, if the left peer decides that it didn't need another connection it terminates the connection. Otherwise, it is legitimate to setup multiple connections since it is equivalent to a second valid request. ii) If the right peer does not accept multiple connections from the same peer then it will reject the request. Note that these decisions in part depend on the ipoib component peeking into the local link address and not on CM. b. REQ first Since its own REQ is outstanding, and as per link address comparison, the left peer accepts then this will be rejected. Vivek > > 2) Left peer rejects > I think that this scenario works. > > Thanks > Dror > >> -----Original Message----- >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] >> Sent: Tuesday, October 25, 2005 11:38 PM >> To: Vandana Rao >> Cc: Dror Goldenberg; 'ipoverib@ietf.org' >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >> >> >> Hi Vandana, >> >> I too agree there is no problem here. >> >> I think you are right if the left peer accepts the >> connection. However, if the reverse was the case, that is the >> left peer rejected the connection since it is found that the >> local address is numerically larger. It will do this >> comparison since its REQ is also outstanding. The right peer >> will not receive any REQ (it is delayed or yet to be >> retransmitted). If it receives the reject it will cancel its >> request. At a later time it will receive the >> delayed/retransmitted request and one connection will be >> established. >> >> Dror, do these two scenarios look right to you? >> >> Vivek >> >> On Thu, 20 Oct 2005, Vandana Rao wrote: >> >>> Hi Dror, >>> I think you are confusing the 2 IDs that come into play in >> this setup. >>> The >>> communication ID is something that the CM entity on a node >> understands but >>> the transaction ID is something that the MAD handling >> entity uses (for >>> retransmission, req/resp matchup etc.). >>> So, in the second slide, there should not be a >> retransmission of the left >>> peer REQ MAD since when the left peer accepts the >> connection, it will cancel >>> its outstanding REQ request and hence the retransmission >> should not take >>> place. >>> -Vandana >>> >>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote: >>>> After giving it some more thought, I think that there is still a >>>> problem. >>>> Please look at the attached pdf. >>>> >>>> In the first slide, everything happens smoothly, the left >> peer knows >>>> that >>>> it has to accept the connection and the right peer knows >> that it has to >>>> decline. >>>> On the 2nd slide, the left peer REQ transmission is >> dropped in the fabric. >>>> It is retransmitted independently by the CM way after a >> channel has already >>>> been established. Now the right side can not really tell >> whether this is a >>>> new channel request or an old "simultaneous connection >> channel". This may >>>> lead to two channels being opened mistakenly. >>>> >>>> I don't believe it can be solved by looking at the >> communication ID. >>>> This >>>> number is just being fabricated by the sender of the REQ >> and is just a >>>> "random number" with respect to what we're trying to determine. >>>> >>>> -Dror >>>> >>>> -----Original Message----- >>>> From: Dror Goldenberg >>>> Sent: Wednesday, October 19, 2005 9:37 PM >>>> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org >>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >>>> >>>> Hi Vandana, >>>> >>>> Thanks for clarifying this. I agree that if IPoiB-CM >> implementation >>>> looks >>>> at the Communication ID it can make those decisions >> correctly. The question >>>> is whether this value is exposed through common CM APIs. >> For all other >>>> purposes, the CM doesn't need to expose the Communication >> ID to the ULP. >>>> However, seems like it's not a big deal to expose this >> value to the ULP. >>>> Anyway, it worth noting this explanation in the RFC so >> that it's clear that >>>> a proper implementation of IPoIB-CM should monitor >> Communication ID as part >>>> of the simultaneous connection decision. >>>> >>>> -Dror >>>> -----Original Message----- >>>> From: Vandana Rao [mailto:vandana.rao@hp.com] >>>> Sent: Wednesday, October 19, 2005 6:40 PM >>>> To: Dror Goldenberg; ipoverib@ietf.org >>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model >>>> >>>> Hi Dror, >>>> The purpose of the MAD transaction ID is to allow one to >> do what you >>>> say >>>> below, i.e. have multiple connection establishment MADs >> between 2 peers. It >>>> is essentially the same as the "channel ID" that you talk >> about below. Its >>>> just that it is a generic channel ID for any MAD >> transactions, not just CM >>>> MADs. >>>> I do not believe there is any issue with existing IB CM in >> this regard. >>>> -Vandana >>>> >>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >>>>> I think that the current model for multiple connections >>>>> establishment >>>>> between >>>>> two peers works well for 1 connection but is broken for >> more than one >>>>> connection. >>>>> Because of cases where CM packets are dropped or travel >> on different VLs, >>>>> they >>>>> might get reordered on the way and observed in a >> different order than were >>>>> originally >>>>> sent. This breaks the current model which assumes that >> the REQs are sent >>>>> and >>>>> processed in order. >>>>> >>>>> I think that we need to think of a model which is robust to CM >>>>> packets >>>>> reordering >>>>> which can be one of: >>>>> - Adding a "channel ID" in the private data. The field >> goes 0,1,2 and on. >>>>> It helps >>>>> synchronizing both peers on which is the channel at >> question now. The >>>>> rest >>>>> of the decision (per channel) of whether to accept or >> reject can be the >>>>> same >>>>> as defined in the draft. >>>>> - Not supporting multiple connections between the same peers. >>>>> >>>>> -Dror >>>>> _______________________________________________ >>>>> IPoverIB mailing list >>>>> IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib >>>> >>> >> > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib From ipoverib-bounces@ietf.org Thu Oct 27 03:04:52 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EV1oi-0006i5-Ez; Thu, 27 Oct 2005 03:04:52 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EV1og-0006gt-3x for ipoverib@megatron.ietf.org; Thu, 27 Oct 2005 03:04:50 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA03113 for ; Thu, 27 Oct 2005 03:04:34 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EV21x-0005C6-W0 for ipoverib@ietf.org; Thu, 27 Oct 2005 03:18:35 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Thu, 27 Oct 2005 09:03:33 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CAEC@mtlexch01.mtl.com> From: Dror Goldenberg To: Vivek Kashyap Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Thu, 27 Oct 2005 09:08:39 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.8 (/) X-Scan-Signature: 6f51ba5266426a53fc06c7d32a3c18b6 Cc: "'ipoverib@ietf.org'" , Vandana Rao X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============2091680401==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============2091680401== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5DAC5.41B84EDC" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5DAC5.41B84EDC Content-Type: text/plain Hi Vivek, I agree with the flows that you describe, and with the fact that the original requester has always the "second chance" to withdraw its request by rejecting the REP. However, in my opinion, giving all this freedom to the implementation can lead to interoperability issues and poor implementations. If an implementation tries to open n connections with a certain peer, it may end up opening more or less than n and that might take few iterations and that all depends on the CM timing and some other "unrelated" parameters - how can you even test it ? I therefore feel that we should provide a robust and deterministic specification of how to establish more than one connection in the RFC or live with a single connection. BTW, this is my opinion in this case, and if folks feel OK with the current draft, then maybe we should move on ? Thanks Dror > -----Original Message----- > From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] > Sent: Wednesday, October 26, 2005 10:33 PM > To: Dror Goldenberg > Cc: Vandana Rao; 'ipoverib@ietf.org' > Subject: RE: [Ipoverib] ipoib-cm multiple connection model > > > On Wed, 26 Oct 2005, Dror Goldenberg wrote: > > > Vivek, > > > > I agree that the two scenarios you mention here are the interesting > > ones. I don't agree about the solution. Let's look at the two cases: > > > > 1) Left peer accepts > > I disagree. You have an assumption that left peer can > > revoke its original connection request - BEFORE transmitting > > the REP that accepts the peer request. I am not sure that > > you can do it with every CM implementation. Even if you can, > > then the HCA can reorder those packets if they are on > > different QPs, and the fabric can reorder them as well if > > they are on different VLs. > > Yes, that has the potential for extra connections..however, > the following will occur if CM checks don't work. > > Left peer accepted. Sent a REP. And of course the REQ was > also sent. The right receiver can receive : > > a. REP first > > It sets up the connection. Then receives a REQ. > > i) If it accepts multiple connections from the same peer: > it will REP with a new QP (connection). > > Now, if the left peer decides that it didn't > need another > connection it terminates the connection. > > Otherwise, it is legitimate to setup multiple > connections > since it is equivalent to a second valid request. > > ii) If the right peer does not accept multiple > connections from the > same peer then it will reject the request. > > Note that these decisions in part depend on the ipoib > component peeking > into the local link address and not on CM. > > > b. REQ first > > Since its own REQ is outstanding, and as per link > address comparison, > the left peer accepts then this will be rejected. > > Vivek > > > > 2) Left peer rejects > > I think that this scenario works. > > > > Thanks > > Dror > > > >> -----Original Message----- > >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] > >> Sent: Tuesday, October 25, 2005 11:38 PM > >> To: Vandana Rao > >> Cc: Dror Goldenberg; 'ipoverib@ietf.org' > >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >> > >> > >> Hi Vandana, > >> > >> I too agree there is no problem here. > >> > >> I think you are right if the left peer accepts the connection. > >> However, if the reverse was the case, that is the left > peer rejected > >> the connection since it is found that the local address is > >> numerically larger. It will do this comparison since its > REQ is also > >> outstanding. The right peer will not receive any REQ (it > is delayed > >> or yet to be retransmitted). If it receives the reject it > will cancel > >> its request. At a later time it will receive the > >> delayed/retransmitted request and one connection will be > >> established. > >> > >> Dror, do these two scenarios look right to you? > >> > >> Vivek > >> > >> On Thu, 20 Oct 2005, Vandana Rao wrote: > >> > >>> Hi Dror, > >>> I think you are confusing the 2 IDs that come into play in > >> this setup. > >>> The > >>> communication ID is something that the CM entity on a node > >> understands but > >>> the transaction ID is something that the MAD handling > >> entity uses (for > >>> retransmission, req/resp matchup etc.). > >>> So, in the second slide, there should not be a > >> retransmission of the left > >>> peer REQ MAD since when the left peer accepts the > >> connection, it will cancel > >>> its outstanding REQ request and hence the retransmission > >> should not take > >>> place. > >>> -Vandana > >>> > >>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote: > >>>> After giving it some more thought, I think that there is still a > >>>> problem. Please look at the attached pdf. > >>>> > >>>> In the first slide, everything happens smoothly, the left > >> peer knows > >>>> that > >>>> it has to accept the connection and the right peer knows > >> that it has to > >>>> decline. > >>>> On the 2nd slide, the left peer REQ transmission is > >> dropped in the fabric. > >>>> It is retransmitted independently by the CM way after a > >> channel has already > >>>> been established. Now the right side can not really tell > >> whether this is a > >>>> new channel request or an old "simultaneous connection > >> channel". This may > >>>> lead to two channels being opened mistakenly. > >>>> > >>>> I don't believe it can be solved by looking at the > >> communication ID. > >>>> This > >>>> number is just being fabricated by the sender of the REQ > >> and is just a > >>>> "random number" with respect to what we're trying to determine. > >>>> > >>>> -Dror > >>>> > >>>> -----Original Message----- > >>>> From: Dror Goldenberg > >>>> Sent: Wednesday, October 19, 2005 9:37 PM > >>>> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org > >>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model > >>>> > >>>> Hi Vandana, > >>>> > >>>> Thanks for clarifying this. I agree that if IPoiB-CM > >> implementation > >>>> looks > >>>> at the Communication ID it can make those decisions > >> correctly. The question > >>>> is whether this value is exposed through common CM APIs. > >> For all other > >>>> purposes, the CM doesn't need to expose the Communication > >> ID to the ULP. > >>>> However, seems like it's not a big deal to expose this > >> value to the ULP. > >>>> Anyway, it worth noting this explanation in the RFC so > >> that it's clear that > >>>> a proper implementation of IPoIB-CM should monitor > >> Communication ID as part > >>>> of the simultaneous connection decision. > >>>> > >>>> -Dror > >>>> -----Original Message----- > >>>> From: Vandana Rao [mailto:vandana.rao@hp.com] > >>>> Sent: Wednesday, October 19, 2005 6:40 PM > >>>> To: Dror Goldenberg; ipoverib@ietf.org > >>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model > >>>> > >>>> Hi Dror, > >>>> The purpose of the MAD transaction ID is to allow one to > >> do what you > >>>> say > >>>> below, i.e. have multiple connection establishment MADs > >> between 2 peers. It > >>>> is essentially the same as the "channel ID" that you talk > >> about below. Its > >>>> just that it is a generic channel ID for any MAD > >> transactions, not just CM > >>>> MADs. > >>>> I do not believe there is any issue with existing IB CM in > >> this regard. > >>>> -Vandana > >>>> > >>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: > >>>>> I think that the current model for multiple connections > >>>>> establishment between > >>>>> two peers works well for 1 connection but is broken for > >> more than one > >>>>> connection. > >>>>> Because of cases where CM packets are dropped or travel > >> on different VLs, > >>>>> they > >>>>> might get reordered on the way and observed in a > >> different order than were > >>>>> originally > >>>>> sent. This breaks the current model which assumes that > >> the REQs are sent > >>>>> and > >>>>> processed in order. > >>>>> > >>>>> I think that we need to think of a model which is robust to CM > >>>>> packets reordering > >>>>> which can be one of: > >>>>> - Adding a "channel ID" in the private data. The field > >> goes 0,1,2 and on. > >>>>> It helps > >>>>> synchronizing both peers on which is the channel at > >> question now. The > >>>>> rest > >>>>> of the decision (per channel) of whether to accept or > >> reject can be the > >>>>> same > >>>>> as defined in the draft. > >>>>> - Not supporting multiple connections between the same peers. > >>>>> > >>>>> -Dror > >>>>> _______________________________________________ > >>>>> IPoverIB mailing list > >>>>> IPoverIB@ietf.org > https://www1.ietf.org/mailman/listinfo/ipoverib > >>>> > >>> > >> > > > ------_=_NextPart_001_01C5DAC5.41B84EDC Content-Type: text/html RE: [Ipoverib] ipoib-cm multiple connection model

Hi Vivek,

I agree with the flows that you describe, and with the fact that the original
requester has always the "second chance" to withdraw its request by
rejecting the REP. However, in my opinion, giving all this freedom to
the implementation can lead to interoperability issues and poor
implementations. If an implementation tries to open n connections with
a certain peer, it may end up opening more or less than n and that
might take few iterations and that all depends on the CM timing and
some other "unrelated" parameters - how can you even test it ?
I therefore feel that we should provide a robust and deterministic
specification of how to establish more than one connection in the
RFC or live with a single connection.

BTW, this is my opinion in this case, and if folks feel OK with the
current draft, then maybe we should move on ?

Thanks
Dror

> -----Original Message-----
> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> Sent: Wednesday, October 26, 2005 10:33 PM
> To: Dror Goldenberg
> Cc: Vandana Rao; 'ipoverib@ietf.org'
> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
>
>
> On Wed, 26 Oct 2005, Dror Goldenberg wrote:
>
> > Vivek,
> >
> > I agree that the two scenarios you mention here are the interesting
> > ones. I don't agree about the solution. Let's look at the two cases:
> >
> > 1) Left peer accepts
> >     I disagree. You have an assumption that left peer can
> >     revoke its original connection request - BEFORE transmitting
> >     the REP that accepts the peer request. I am not sure that
> >     you can do it with every CM implementation. Even if you can,
> >     then the HCA can reorder those packets if they are on
> >     different QPs, and the fabric can reorder them as well if
> >     they are on different VLs.
>
> Yes, that has the potential for extra connections..however,
> the following will occur if CM checks don't work.
>
> Left peer accepted. Sent a REP. And of course the REQ was
> also sent. The right receiver can receive :
>
>       a. REP first
>
>       It sets up the connection. Then receives a REQ.
>
>       i) If it accepts multiple connections from the same peer:
>                it will  REP with a new QP (connection).
>
>               Now, if the left peer decides that it didn't
> need another
>               connection it terminates the connection.
>
>               Otherwise, it is legitimate to setup multiple
> connections
>               since it is equivalent to a second valid request.
>
>       ii) If the right peer does not accept multiple
> connections from the
>       same peer then it will reject the request.
>
>       Note that these decisions in part depend on the ipoib
> component peeking
>       into the local link address and not on CM.
>
>
>       b. REQ first
>
>       Since its own REQ is outstanding, and as per link
> address comparison,
>       the left peer accepts then this will be rejected.
>
> Vivek
> >
> > 2) Left peer rejects
> >     I think that this scenario works.
> >
> > Thanks
> > Dror
> >
> >> -----Original Message-----
> >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com]
> >> Sent: Tuesday, October 25, 2005 11:38 PM
> >> To: Vandana Rao
> >> Cc: Dror Goldenberg; 'ipoverib@ietf.org'
> >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>
> >>
> >> Hi Vandana,
> >>
> >> I too agree there is no problem here.
> >>
> >> I think you are right if the left peer accepts the connection.
> >> However, if the reverse was the case, that is the left
> peer rejected
> >> the connection since it is found that the local address is
> >> numerically larger. It will do this comparison since its
> REQ is also
> >> outstanding.  The right peer will not receive any REQ  (it
> is delayed
> >> or yet to be retransmitted). If it receives the reject it
> will cancel
> >> its request. At a later time it will receive the
> >> delayed/retransmitted request and one connection will be
> >> established.
> >>
> >> Dror, do these two scenarios look right to you?
> >>
> >> Vivek
> >>
> >> On Thu, 20 Oct 2005, Vandana Rao wrote:
> >>
> >>> Hi Dror,
> >>> I think you are confusing the 2 IDs that come into play in
> >> this setup.
> >>> The
> >>> communication ID is something that the CM entity on a node
> >> understands but
> >>> the transaction ID is something that the MAD handling
> >> entity uses (for
> >>> retransmission, req/resp matchup etc.).
> >>> So, in the second slide, there should not be a
> >> retransmission of the left
> >>> peer REQ MAD since when the left peer accepts the
> >> connection, it will cancel
> >>> its outstanding REQ request and hence the retransmission
> >> should not take
> >>> place.
> >>> -Vandana
> >>>
> >>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
> >>>> After giving it some more thought, I think that there is still a
> >>>> problem. Please look at the attached pdf.
> >>>>
> >>>> In the first slide, everything happens smoothly, the left
> >> peer knows
> >>>> that
> >>>> it has to accept the connection and the right peer knows
> >> that it has to
> >>>> decline.
> >>>> On the 2nd slide, the left peer REQ transmission is
> >> dropped in the fabric.
> >>>> It is retransmitted independently by the CM way after a
> >> channel has already
> >>>> been established. Now the right side can not really tell
> >> whether this is a
> >>>> new channel request or an old "simultaneous connection
> >> channel". This may
> >>>> lead to two channels being opened mistakenly.
> >>>>
> >>>> I don't believe it can be solved by looking at the
> >> communication ID.
> >>>> This
> >>>> number is just being fabricated by the sender of the REQ
> >> and is just a
> >>>> "random number" with respect to what we're trying to determine.
> >>>>
> >>>> -Dror
> >>>>
> >>>> -----Original Message-----
> >>>> From: Dror Goldenberg
> >>>> Sent: Wednesday, October 19, 2005 9:37 PM
> >>>> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
> >>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model
> >>>>
> >>>> Hi Vandana,
> >>>>
> >>>> Thanks for clarifying this. I agree that if IPoiB-CM
> >> implementation
> >>>> looks
> >>>> at the Communication ID it can make those decisions
> >> correctly. The question
> >>>> is whether this value is exposed through common CM APIs.
> >> For all other
> >>>> purposes, the CM doesn't need to expose the Communication
> >> ID to the ULP.
> >>>> However, seems like it's not a big deal to expose this
> >> value to the ULP.
> >>>> Anyway, it worth noting this explanation in the RFC so
> >> that it's clear that
> >>>> a proper implementation of IPoIB-CM should monitor
> >> Communication ID as part
> >>>> of the simultaneous connection decision.
> >>>>
> >>>> -Dror
> >>>> -----Original Message-----
> >>>> From: Vandana Rao [mailto:vandana.rao@hp.com]
> >>>> Sent: Wednesday, October 19, 2005 6:40 PM
> >>>> To: Dror Goldenberg; ipoverib@ietf.org
> >>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model
> >>>>
> >>>> Hi Dror,
> >>>> The purpose of the MAD transaction ID is to allow one to
> >> do what you
> >>>> say
> >>>> below, i.e. have multiple connection establishment MADs
> >> between 2 peers. It
> >>>> is essentially the same as the "channel ID" that you talk
> >> about below. Its
> >>>> just that it is a generic channel ID for any MAD
> >> transactions, not just CM
> >>>> MADs.
> >>>> I do not believe there is any issue with existing IB CM in
> >> this regard.
> >>>> -Vandana
> >>>>
> >>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
> >>>>> I think that the current model for multiple connections
> >>>>> establishment between
> >>>>> two peers works well for 1 connection but is broken for
> >> more than one
> >>>>> connection.
> >>>>> Because of cases where CM packets are dropped or travel
> >> on different VLs,
> >>>>> they
> >>>>> might get reordered on the way and observed in a
> >> different order than were
> >>>>> originally
> >>>>> sent. This breaks the current model which assumes that
> >> the REQs are sent
> >>>>> and
> >>>>> processed in order.
> >>>>>
> >>>>> I think that we need to think of a model which is robust to CM
> >>>>> packets reordering
> >>>>> which can be one of:
> >>>>> - Adding a "channel ID" in the private data. The field
> >> goes 0,1,2 and on.
> >>>>> It helps
> >>>>>    synchronizing both peers on which is the channel at
> >> question now. The
> >>>>> rest
> >>>>>    of the decision (per channel) of whether to accept or
> >> reject can be the
> >>>>> same
> >>>>>    as defined in the draft.
> >>>>> - Not supporting multiple connections between the same peers.
> >>>>>
> >>>>> -Dror
> >>>>> _______________________________________________
> >>>>> IPoverIB mailing list
> >>>>> IPoverIB@ietf.org
> https://www1.ietf.org/mailman/listinfo/ipoverib
> >>>>
> >>>
> >>
> >
>

------_=_NextPart_001_01C5DAC5.41B84EDC-- --===============2091680401== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============2091680401==-- From ipoverib-bounces@ietf.org Thu Oct 27 03:04:54 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EV1ok-0006iX-50; Thu, 27 Oct 2005 03:04:54 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EV1og-0006gu-4L for ipoverib@megatron.ietf.org; Thu, 27 Oct 2005 03:04:51 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA03111 for ; Thu, 27 Oct 2005 03:04:33 -0400 (EDT) Received: from [194.90.237.34] (helo=mtlex01.yok.mtl.com) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EV21x-0005C5-W8 for ipoverib@ietf.org; Thu, 27 Oct 2005 03:18:35 -0400 Received: by mtlex01.yok.mtl.com with Internet Mail Service (5.5.2653.19) id ; Thu, 27 Oct 2005 09:03:33 +0200 Message-ID: <6AB138A2AB8C8E4A98B9C0C3D52670E335CAEB@mtlexch01.mtl.com> From: Dror Goldenberg To: Vandana Rao , Dror Goldenberg , "'ipoverib@ietf.org'" Subject: RE: [Ipoverib] ipoib-cm multiple connection model Date: Thu, 27 Oct 2005 09:08:37 +0200 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) X-Spam-Score: 0.1 (/) X-Scan-Signature: 14278aea5bdd1edf35ec09ffb7b61f9d Cc: X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Content-Type: multipart/mixed; boundary="===============0258544351==" Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0258544351== Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5DAC5.40B46E8A" This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C5DAC5.40B46E8A Content-Type: text/plain -----Original Message----- From: Vandana Rao [mailto:vandana.rao@hp.com] Sent: Wednesday, October 26, 2005 8:35 PM To: Dror Goldenberg; 'ipoverib@ietf.org' Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Dror, You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP. But the point is that those MADs in question belong to different transactions, one is REP for transaction 1 and other is REQ for transaction 2. Therefore the CM need not guarantee any ordering between them. If not, there is potential for the state machines to get messed up. -Vandana At 12:20 AM 10/26/2005, Dror Goldenberg wrote: Hi Vandana, The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases: 1) The CM uses more than one QP, in which case the HCA can reorder those packets 2) The CM uses different SL/VL, in which case the fabric can reorder the packets 3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU. My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required. -Dror -----Original Message----- From: Vandana Rao [ mailto:vandana.rao@hp.com ] Sent: Thursday, October 20, 2005 7:25 PM To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org' Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Dror, I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.). So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place. -Vandana At 12:39 AM 10/20/2005, Dror Goldenberg wrote: After giving it some more thought, I think that there is still a problem. Please look at the attached pdf. In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline. On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly. I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine. -Dror -----Original Message----- From: Dror Goldenberg Sent: Wednesday, October 19, 2005 9:37 PM To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org Subject: RE: [Ipoverib] ipoib-cm multiple connection model Hi Vandana, Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP. Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision. -Dror -----Original Message----- From: Vandana Rao [ mailto:vandana.rao@hp.com ] Sent: Wednesday, October 19, 2005 6:40 PM To: Dror Goldenberg; ipoverib@ietf.org Subject: Re: [Ipoverib] ipoib-cm multiple connection model Hi Dror, The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs. I do not believe there is any issue with existing IB CM in this regard. -Vandana At 03:11 AM 10/19/2005, Dror Goldenberg wrote: I think that the current model for multiple connections establishment between two peers works well for 1 connection but is broken for more than one connection. Because of cases where CM packets are dropped or travel on different VLs, they might get reordered on the way and observed in a different order than were originally sent. This breaks the current model which assumes that the REQs are sent and processed in order. I think that we need to think of a model which is robust to CM packets reordering which can be one of: - Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps synchronizing both peers on which is the channel at question now. The rest of the decision (per channel) of whether to accept or reject can be the same as defined in the draft. - Not supporting multiple connections between the same peers. -Dror _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib ------_=_NextPart_001_01C5DAC5.40B46E8A Content-Type: text/html Message
 
-----Original Message-----
From: Vandana Rao [mailto:vandana.rao@hp.com]
Sent: Wednesday, October 26, 2005 8:35 PM
To: Dror Goldenberg; 'ipoverib@ietf.org'
Subject: RE: [Ipoverib] ipoib-cm multiple connection model

Hi Dror,
You are absolutely right. It is upto the CM implementation (if it is using multiple QPs for its communication) to ensure that the MADs belonging to the same transaction (by that I mean all the MAD's to setup a connection) end up on the same UD QP.  
 
But the point is that those MADs in question belong to different transactions, one is REP for transaction 1 and other is REQ for transaction 2. Therefore the CM need not guarantee any ordering between them.
 
 If not, there is potential for the state machines to get messed up.
-Vandana

At 12:20 AM 10/26/2005, Dror Goldenberg wrote:
Hi Vandana,
 
The ability to revoke a request is a CM API issue that may or may not be present in an implementation. Assuming it is present, you might be late to revoke, e.g. the CM has already scheduled the REQ to be retransmitted. In which case, you have a very strong assumption that the retransmitted REQ will go out to the wire BEFORE the REP will. This assumption may break in the following cases:
1) The CM uses more than one QP, in which case the HCA can reorder those packets
2) The CM uses different SL/VL, in which case the fabric can reorder the packets
3) The CM may choose to internally queue the packets and then submit them to the HCA. This internal queuing mechanism can reorder packets, e.g. when you have a queue per CPU.
 
My feeling is that relying on packet ordering between different unrelated streams won't be robust, and can cause more connections to be established than were actually required.
 
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Thursday, October 20, 2005 7:25 PM
To: Dror Goldenberg; Dror Goldenberg; 'ipoverib@ietf.org'
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
I think you are confusing the 2 IDs that come into play in this setup. The communication ID is something that the CM entity on a node understands but the transaction ID is something that the MAD handling entity uses (for retransmission, req/resp matchup etc.).
 So, in the second slide, there should not be a retransmission of the left peer REQ MAD since when the left peer accepts the connection, it will cancel its outstanding REQ request and hence the retransmission should not take place.
-Vandana

At 12:39 AM 10/20/2005, Dror Goldenberg wrote:
After giving it some more thought, I think that there is still a problem. Please look at the attached pdf.

 
In the first slide, everything happens smoothly, the left peer knows that it has to accept the connection and the right peer knows that it has to decline.
On the 2nd slide, the left peer REQ transmission is dropped in the fabric. It is retransmitted independently by the CM way after a channel has already been established. Now the right side can not really tell whether this is a new channel request or an old "simultaneous connection channel". This may lead to two channels being opened mistakenly.

 
I don't believe it can be solved by looking at the communication ID. This number is just being fabricated by the sender of the REQ and is just a "random number" with respect to what we're trying to determine.

 
-Dror
 
-----Original Message-----
From: Dror Goldenberg
Sent: Wednesday, October 19, 2005 9:37 PM
To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org
Subject: RE: [Ipoverib] ipoib-cm multiple connection model
Hi Vandana,
Thanks for clarifying this. I agree that if IPoiB-CM implementation looks at the Communication ID it can make those decisions correctly. The question is whether this value is exposed through common CM APIs. For all other purposes, the CM doesn't need to expose the Communication ID to the ULP. However, seems like it's not a big deal to expose this value to the ULP.
Anyway, it worth noting this explanation in the RFC so that it's clear that a proper implementation of IPoIB-CM should monitor Communication ID as part of the simultaneous connection decision.
-Dror
-----Original Message-----
From: Vandana Rao [ mailto:vandana.rao@hp.com]
Sent: Wednesday, October 19, 2005 6:40 PM
To: Dror Goldenberg; ipoverib@ietf.org
Subject: Re: [Ipoverib] ipoib-cm multiple connection model
Hi Dror,
The purpose of the MAD transaction ID is to allow one to do what you say below, i.e. have multiple connection establishment MADs between 2 peers. It is essentially the same as the "channel ID" that you talk about below. Its just that it is a generic channel ID for any MAD transactions, not just CM MADs.
I do not believe there is any issue with existing IB CM in this regard.
-Vandana
At 03:11 AM 10/19/2005, Dror Goldenberg wrote:
I think that the current model for multiple connections establishment between
two peers works well for 1 connection but is broken for more than one connection.
Because of cases where CM packets are dropped or travel on different VLs, they
might get reordered on the way and observed in a different order than were originally
sent. This breaks the current model which assumes that the REQs are sent and
processed in order.
I think that we need to think of a model which is robust to CM packets reordering
which can be one of:
- Adding a "channel ID" in the private data. The field goes 0,1,2 and on. It helps
   synchronizing both peers on which is the channel at question now. The rest
   of the decision (per channel) of whether to accept or reject can be the same
   as defined in the draft.
- Not supporting multiple connections between the same peers.
-Dror
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
_______________________________________________
IPoverIB mailing list
IPoverIB@ietf.org
https://www1.ietf.org/mailman/listinfo/ipoverib
------_=_NextPart_001_01C5DAC5.40B46E8A-- --===============0258544351== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib --===============0258544351==-- From ipoverib-bounces@ietf.org Fri Oct 28 01:59:47 2005 Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EVNHG-0002Ay-Sx; Fri, 28 Oct 2005 01:59:46 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1EVNGC-00022d-RI for ipoverib@megatron.ietf.org; Fri, 28 Oct 2005 01:58:45 -0400 Received: from ietf-mx.ietf.org (ietf-mx [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id BAA23134 for ; Fri, 28 Oct 2005 01:58:19 -0400 (EDT) Received: from e36.co.us.ibm.com ([32.97.110.154]) by ietf-mx.ietf.org with esmtp (Exim 4.43) id 1EVNTS-0001fv-5R for ipoverib@ietf.org; Fri, 28 Oct 2005 02:12:22 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e36.co.us.ibm.com (8.12.11/8.12.11) with ESMTP id j9S5wASc027010 for ; Fri, 28 Oct 2005 01:58:10 -0400 Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by westrelay02.boulder.ibm.com (8.12.10/NCO/VERS6.7) with ESMTP id j9S5wA4g535604 for ; Thu, 27 Oct 2005 23:58:10 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.12.11/8.13.3) with ESMTP id j9S5wAaq018342 for ; Thu, 27 Oct 2005 23:58:10 -0600 Received: from sig-9-48-50-253.mts.ibm.com (sig-9-48-50-253.mts.ibm.com [9.48.50.253]) by d03av02.boulder.ibm.com (8.12.11/8.12.11) with ESMTP id j9S5w8iW018314; Thu, 27 Oct 2005 23:58:09 -0600 Date: Thu, 27 Oct 2005 22:47:16 -0700 (PDT) From: Vivek Kashyap X-X-Sender: kashyapv@localhost.localdomain To: Dror Goldenberg Subject: RE: [Ipoverib] ipoib-cm multiple connection model In-Reply-To: <6AB138A2AB8C8E4A98B9C0C3D52670E335CAEC@mtlexch01.mtl.com> Message-ID: References: <6AB138A2AB8C8E4A98B9C0C3D52670E335CAEC@mtlexch01.mtl.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Score: 0.0 (/) X-Scan-Signature: f8ee348dcc4be4a59bc395f7cd6343ad Cc: "'ipoverib@ietf.org'" , Vandana Rao X-BeenThere: ipoverib@ietf.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IP over InfiniBand WG Discussion List List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: ipoverib-bounces@ietf.org Errors-To: ipoverib-bounces@ietf.org Hi Dror, On Thu, 27 Oct 2005, Dror Goldenberg wrote: > Hi Vivek, > > I agree with the flows that you describe, and with the fact that the > original > requester has always the "second chance" to withdraw its request by > rejecting the REP. However, in my opinion, giving all this freedom to > the implementation can lead to interoperability issues and poor > implementations. If an implementation tries to open n connections with > a certain peer, it may end up opening more or less than n and that > might take few iterations and that all depends on the CM timing and > some other "unrelated" parameters - how can you even test it ? > I therefore feel that we should provide a robust and deterministic > specification of how to establish more than one connection in the > RFC or live with a single connection. I'd like to delve into this a bit more to be sure. As per your/Vandana's comments, the left peer will cancel the request. Isn't this is a 'conscious' decision by the ULP? The ULP would then remove the its internal object that it would associate with this IB connection. If it was successful in cancelling the REQ then there is only one connection. If, however, as you pointed out it is unable to, it might receive a REP *if* the remote peer accepts multiple connections. The left peer's ulp however will not find any object associated with the private data and the connection request (there will be another one already existing though) and can use that to indicate rejection. Does the above sound right? Vivek > > BTW, this is my opinion in this case, and if folks feel OK with the > current draft, then maybe we should move on ? > > Thanks > Dror > >> -----Original Message----- >> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] >> Sent: Wednesday, October 26, 2005 10:33 PM >> To: Dror Goldenberg >> Cc: Vandana Rao; 'ipoverib@ietf.org' >> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >> >> >> On Wed, 26 Oct 2005, Dror Goldenberg wrote: >> >>> Vivek, >>> >>> I agree that the two scenarios you mention here are the interesting >>> ones. I don't agree about the solution. Let's look at the two cases: >>> >>> 1) Left peer accepts >>> I disagree. You have an assumption that left peer can >>> revoke its original connection request - BEFORE transmitting >>> the REP that accepts the peer request. I am not sure that >>> you can do it with every CM implementation. Even if you can, >>> then the HCA can reorder those packets if they are on >>> different QPs, and the fabric can reorder them as well if >>> they are on different VLs. >> >> Yes, that has the potential for extra connections..however, >> the following will occur if CM checks don't work. >> >> Left peer accepted. Sent a REP. And of course the REQ was >> also sent. The right receiver can receive : >> >> a. REP first >> >> It sets up the connection. Then receives a REQ. >> >> i) If it accepts multiple connections from the same peer: >> it will REP with a new QP (connection). >> >> Now, if the left peer decides that it didn't >> need another >> connection it terminates the connection. >> >> Otherwise, it is legitimate to setup multiple >> connections >> since it is equivalent to a second valid request. >> >> ii) If the right peer does not accept multiple >> connections from the >> same peer then it will reject the request. >> >> Note that these decisions in part depend on the ipoib >> component peeking >> into the local link address and not on CM. >> >> >> b. REQ first >> >> Since its own REQ is outstanding, and as per link >> address comparison, >> the left peer accepts then this will be rejected. >> >> Vivek >>> >>> 2) Left peer rejects >>> I think that this scenario works. >>> >>> Thanks >>> Dror >>> >>>> -----Original Message----- >>>> From: Vivek Kashyap [mailto:kashyapv@us.ibm.com] >>>> Sent: Tuesday, October 25, 2005 11:38 PM >>>> To: Vandana Rao >>>> Cc: Dror Goldenberg; 'ipoverib@ietf.org' >>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >>>> >>>> >>>> Hi Vandana, >>>> >>>> I too agree there is no problem here. >>>> >>>> I think you are right if the left peer accepts the connection. >>>> However, if the reverse was the case, that is the left >> peer rejected >>>> the connection since it is found that the local address is >>>> numerically larger. It will do this comparison since its >> REQ is also >>>> outstanding. The right peer will not receive any REQ (it >> is delayed >>>> or yet to be retransmitted). If it receives the reject it >> will cancel >>>> its request. At a later time it will receive the >>>> delayed/retransmitted request and one connection will be >>>> established. >>>> >>>> Dror, do these two scenarios look right to you? >>>> >>>> Vivek >>>> >>>> On Thu, 20 Oct 2005, Vandana Rao wrote: >>>> >>>>> Hi Dror, >>>>> I think you are confusing the 2 IDs that come into play in >>>> this setup. >>>>> The >>>>> communication ID is something that the CM entity on a node >>>> understands but >>>>> the transaction ID is something that the MAD handling >>>> entity uses (for >>>>> retransmission, req/resp matchup etc.). >>>>> So, in the second slide, there should not be a >>>> retransmission of the left >>>>> peer REQ MAD since when the left peer accepts the >>>> connection, it will cancel >>>>> its outstanding REQ request and hence the retransmission >>>> should not take >>>>> place. >>>>> -Vandana >>>>> >>>>> At 12:39 AM 10/20/2005, Dror Goldenberg wrote: >>>>>> After giving it some more thought, I think that there is still a >>>>>> problem. Please look at the attached pdf. >>>>>> >>>>>> In the first slide, everything happens smoothly, the left >>>> peer knows >>>>>> that >>>>>> it has to accept the connection and the right peer knows >>>> that it has to >>>>>> decline. >>>>>> On the 2nd slide, the left peer REQ transmission is >>>> dropped in the fabric. >>>>>> It is retransmitted independently by the CM way after a >>>> channel has already >>>>>> been established. Now the right side can not really tell >>>> whether this is a >>>>>> new channel request or an old "simultaneous connection >>>> channel". This may >>>>>> lead to two channels being opened mistakenly. >>>>>> >>>>>> I don't believe it can be solved by looking at the >>>> communication ID. >>>>>> This >>>>>> number is just being fabricated by the sender of the REQ >>>> and is just a >>>>>> "random number" with respect to what we're trying to determine. >>>>>> >>>>>> -Dror >>>>>> >>>>>> -----Original Message----- >>>>>> From: Dror Goldenberg >>>>>> Sent: Wednesday, October 19, 2005 9:37 PM >>>>>> To: 'Vandana Rao'; Dror Goldenberg; ipoverib@ietf.org >>>>>> Subject: RE: [Ipoverib] ipoib-cm multiple connection model >>>>>> >>>>>> Hi Vandana, >>>>>> >>>>>> Thanks for clarifying this. I agree that if IPoiB-CM >>>> implementation >>>>>> looks >>>>>> at the Communication ID it can make those decisions >>>> correctly. The question >>>>>> is whether this value is exposed through common CM APIs. >>>> For all other >>>>>> purposes, the CM doesn't need to expose the Communication >>>> ID to the ULP. >>>>>> However, seems like it's not a big deal to expose this >>>> value to the ULP. >>>>>> Anyway, it worth noting this explanation in the RFC so >>>> that it's clear that >>>>>> a proper implementation of IPoIB-CM should monitor >>>> Communication ID as part >>>>>> of the simultaneous connection decision. >>>>>> >>>>>> -Dror >>>>>> -----Original Message----- >>>>>> From: Vandana Rao [mailto:vandana.rao@hp.com] >>>>>> Sent: Wednesday, October 19, 2005 6:40 PM >>>>>> To: Dror Goldenberg; ipoverib@ietf.org >>>>>> Subject: Re: [Ipoverib] ipoib-cm multiple connection model >>>>>> >>>>>> Hi Dror, >>>>>> The purpose of the MAD transaction ID is to allow one to >>>> do what you >>>>>> say >>>>>> below, i.e. have multiple connection establishment MADs >>>> between 2 peers. It >>>>>> is essentially the same as the "channel ID" that you talk >>>> about below. Its >>>>>> just that it is a generic channel ID for any MAD >>>> transactions, not just CM >>>>>> MADs. >>>>>> I do not believe there is any issue with existing IB CM in >>>> this regard. >>>>>> -Vandana >>>>>> >>>>>> At 03:11 AM 10/19/2005, Dror Goldenberg wrote: >>>>>>> I think that the current model for multiple connections >>>>>>> establishment between >>>>>>> two peers works well for 1 connection but is broken for >>>> more than one >>>>>>> connection. >>>>>>> Because of cases where CM packets are dropped or travel >>>> on different VLs, >>>>>>> they >>>>>>> might get reordered on the way and observed in a >>>> different order than were >>>>>>> originally >>>>>>> sent. This breaks the current model which assumes that >>>> the REQs are sent >>>>>>> and >>>>>>> processed in order. >>>>>>> >>>>>>> I think that we need to think of a model which is robust to CM >>>>>>> packets reordering >>>>>>> which can be one of: >>>>>>> - Adding a "channel ID" in the private data. The field >>>> goes 0,1,2 and on. >>>>>>> It helps >>>>>>> synchronizing both peers on which is the channel at >>>> question now. The >>>>>>> rest >>>>>>> of the decision (per channel) of whether to accept or >>>> reject can be the >>>>>>> same >>>>>>> as defined in the draft. >>>>>>> - Not supporting multiple connections between the same peers. >>>>>>> >>>>>>> -Dror >>>>>>> _______________________________________________ >>>>>>> IPoverIB mailing list >>>>>>> IPoverIB@ietf.org >> https://www1.ietf.org/mailman/listinfo/ipoverib >>>>>> >>>>> >>>> >>> >> > _______________________________________________ IPoverIB mailing list IPoverIB@ietf.org https://www1.ietf.org/mailman/listinfo/ipoverib