Subject: Re: [Rum] [EXT] Re: Real-time text in WebRTC is discussed in mmusic - is recvonly etc support desired?

From: Gunnar Hellström <[email-masked]>
Date: Sun, 1 Sep 2019 23:26:51 +0200

I would like a comment from RUM if it is desirable to have support for 
the direction attributes : sendrecv, sendonly, recvonly, inactive,  for 
RTT streams?

It might be for control of muting etc in multi-party situations, but of 
course also in other situations.

Regards

Gunnar

Den 2019-08-31 kl. 22:47, skrev Gunnar Hellström:
> Hi John,
>
> Thanks for a good base for discussion.
>
> See inline,
>
> Den 2019-08-30 kl. 17:12, skrev Paul Kyzivat:
>> John,
>>
>> On 8/30/19 7:32 AM, John Martin wrote:
>>> All,
>>>
>>>   $0.02 on RTT:
>>
>> Thanks for joining in! I don't have much knowledge of (and no 
>> experience using) RTT so I will mostly watch from the sidelines as 
>> long as others join in.
>>
>>     Thanks,
>>     Paul
>>
>> (BTW, I don't know how you formatted your message, but in my mail 
>> reader (Thunderbird) none of your lines wrapped, requiring horizontal 
>> scrolling to read them.)
>>
>>> RTT in General
>>>
>>> --------------
>>>
>>> ISTM there are a few options for RTT in RUM, assuming the WebRTC 
>>> media framework is adopted (as seems likely):
>>>
>>> 1.Use existing browser support for SCTP as recommended by the some 
>>> of the WebRTC IETF WG participants. This was discussed as part of 
>>> the IVC WG and would require and SCTP data channel to be negotiated 
>>> and to encapsulate the RTP (4103) which then in turn encapsulate 
>>> T.140 RTT. This solution would require gateways in the VRS provider 
>>> networks, RTP/RTCP userspace (Javascript) layers in the browser and 
>>> a T.140 RTT layer/UI also written in userspace in the browser.
>
> The current work discussed in mmusic  about 
> draft-holmberg-mmusic-t140-usage-data-channel  specifies T.140 
> real-time text transported directly in a "reliable" WebRTC data 
> channel, identified with subprotocol "t140". Would that not be 
> suitable for a WebRTC based RUM? The data payload will be the same as 
> carried in RFC 4103 if used without redundancy. You mention that 
> embedded RTP with RFC 4103 would be used as data also in the data 
> channel. That sounds complicated. What would you gain?
>
> I hope you also follow the discussion in mmusic.
>
>>>
>>> 2.Wait for WebRTC to support lower level data channels. This work is 
>>> being discussed in the WebRTC community, but might be a case of “not 
>>> holding our breath”. A lower level data channel would allow RTP/RTCP 
>>> then T.140 to be negotiated in the SDP without the need for SCTP and 
>>> associated gateway needs.
> Since the RTCWEB WG is closed now in IETF, this sounds as a 7 year 
> uncertain plan.
>>>
>>>
>>> 3.As a stop-gap and simplification, support SIP MESSAGE as already 
>>> provided for by the browsers and many endpoints, including WebRTC 
>>> endpoints.
> SCTP data channels are supported as well, and you can just agree to 
> start using it with a private subprotocol temporarily among RUMs until 
> there is a standard. So why use SIP MESSAGE?
>>>
>>>
>>> We should not forget that RTCP must be implemented along with RTP.
>>>
>>> Conferencing and RTT
>>>
>>> --------------------
>>>
>>> RTT in a conference is not currently defined and does not work using 
>>> currently defined standards (I must admit to being being on mmusic 
>>> so perhaps there are solutions being recommended there). Discussions 
>>> on this have been ongoing for nearly 10 years (we didn’t solve it 
>>> then Gunnar and I think the problems are still the same). For RTT to 
>>> be useful in a conference there needs to be a few problems solved.
>
> If you do it with multiple streams sent to the clients, then there are 
> no problems. Therefore there was hesitation when we discussed to bring 
> up the topic in IETF.
>
> For mixing into one stream, there are problems. You can do it with 
> normal RTP and conferencing specifications but get some limitations. 
> You can select between a presentation planned by the mixer, or use 
> CSRC in RTP to indicate the source of primary data in each packet and 
> let the client plan the presentation. Both work and do not really need 
> any further standardisation, but would benefit from good best practice 
> specifications or even a standard for solving some problems.
>
> Planning and formatting the presentation in the mixer has the 
> limitation that only the local user and one at a time of the remote 
> users text can be presented in real-time. Text from the others need to 
> be buffered until the currently presenting remote participant makes 
> something so that giving turn to next in queue is suitable. Switching 
> turn can be made automatically, based on completed phrase or sentence 
> or message or inactivity or allowed maximum time by the presenting 
> party. Identification about who is sending which text needs to be 
> formatted by the mixer and is likely best in form of an IRC style 
> label when the source is switched. This works for a call where users 
> are reasonably well-behaved in taking turns. There are cases where the 
> presentation in one column with sources in labels is not what the user 
> wants.
>
> Using just one stream, and mark by CSRC who contributed to primary 
> data in each packet works as long as no packet was lost. The receiver 
> can make a better adapted presentation for presentation preferences by 
> the user. And text can be presented in real-time from more than two 
> users at the same time.
>
> But in order to use the redundant RFC 4103 data to recover text in 
> case of packet loss, then also the redundant data needs to be marked 
> with source. And for that, a convention is needed and standardised. A 
> couple of variants are possible with different pros and cons. Losing 
> so many packets that a missing text marker needs to be inserted also 
> has its problems that needs a specified solution.
>
>>>
>>> 1.Multi-stream support: Brian, whilst you’re right that most 
>>> conference systems are point-to-point for signalling they are more 
>>> typically point-to-multi-point for media. The conference bridge 
>>> usually handles the multiplexing of the streams, but they are 
>>> usually now multiple streams. RTT conferencing needs to have the 
>>> same support. We should assume conferencing is supported by way of 
>>> SFU not MCU – that’s how WebRTC handles conferencing and we should 
>>> assume the same for RTT.
> Yes, that is now the main path for 
> draft-holmberg-mmusic-t140-usage-data-channel, and I think that is 
> good. That is also possible without any further standardization for 
> RTP based RTT.
>>>
>>> 2.Participant identification: streams from the conference bridge 
>>> (MCU/SFU) must be tagged with the sender so the UI can signal to the 
>>> user who sent that text (and reconstruct the stream correctly).
> Yes. Use CSRC / CNAME for RTP and  stream ID / label / session 
> information for WebRTC data channel.
>>>
>>> 3.UI adaptions: Modern WebRTC based multi-party conferences have 
>>> adapted UIs to display multiple video streams. UI’s are adapted to 
>>> show the video participants however the user wants them to be 
>>> displayed (single full screen, tiled, voice activated etc). We 
>>> should expect no less from RTT conference enabled endpoints. It 
>>> should be a user decision (if their UI supports it) as to whether 
>>> they see line-by-line or RTT rendering of the text conversation.
> Yes. If you want a one column presentation, a reasonable presentation 
> is to automatically assign one remote party at a time to be presented 
> in real-time and present the others phrase by phrase, and 
> automatically switch who is presented in real-time based on suitable 
> criteria.
>>>
>>> 4.Control of how many simultaneous participants can text and how 
>>> that is presented to the user needs to be explored. Multiple 
>>> insertion points in the UI is one approach (though rapidly breaks 
>>> down as the number of users increases). Line-by-line is another. 
>>> Facilitated GA is another (the Go Ahead concept has been used for a 
>>> long time in text conversations and may be appropriate in large RTT 
>>> conferences to control overload).
>>>
>>> And a final FWIW: I think the utility of RTT in a conference quickly 
>>> breaks down as the number of participants increases. It is 
>>> completely unacceptable to have a single RTT stream presented to the 
>>> user and an unmodified UI. You might as well revert to SIP MESSAGE 
>>> and something like XMPP, and that could also be an option for either 
>>> the short term or for endpoints that do not support multi-stream RTT.
>
> In some conferences it might be desirable with one dominating RTT 
> source: a written interpretation of what is said, and then 
> occasionally another participant gets the floor and types something . 
> It is very valuable if that mode of conferencing is available without 
> hazzle in general multi-party conference systems. The utility does not 
> break down in that situation.
>
> It seems to me that aiming at multi-stream RTT support is to prefer.
>
>
> Regards
>
> Gunnar
>
>>>
>>> John
>>>
>>> Brian said: we agreed we were using 4103 in RUM.
>>>
>>> Good toknow. But is there not a hope to be able to even use browser
>>>
>>> technology for the implementation. The browsers only support RTP for
>>>
>>> audio and video. Can you make RTP-based RTT work in that environment 
>>> and
>>>
>>> encrypted with the security method specified for WebRTC?
>>>
>>> See a bit more below:
>>>
>>> Den 2019-08-29 kl. 21:59, skrev Malloy, Jim:
>>>
>>>> “line at a time” is a user interface issue – RUM is defining a machine 
>>>
>>>> interface.  How it’s presented to the user is out of scope.
>>>
>>>>
>>>
>>> Well, the control and coding must be made so that it is possible to 
>>> make
>>>
>>> a good presentation. Transmission only line-by-line would not suit the
>>>
>>> intention of RTT. It is hard to make a good RTT presentation with a
>>>
>>> client that is only expecting text from one other participant and a
>>>
>>> mixer that has the task to send text from many sources combined in one
>>>
>>> stream. The best result is usable but not nice. And it will be aimed at
>>>
>>> just one way to present RTT. No freedom to plan the presentation
>>>
>>> according to user preferences.
>>>
>>> A receiver may, if there is a desire for such functionality, store
>>>
>>> received text until a received message is complete. I do not think that
>>>
>>> it is to prefer in any situation.
>>>
>>> Regards
>>>
>>> Gunnar
>>>
>>>> --Jim
>>>
>>>>
>>>
>>>> *From:* Rum <[email-masked]> 
>>>> <mailto:[email-masked]&gt>; *On Behalf Of * 
>>> Brian Rosen
>>>
>>>> *Sent:* Thursday, August 29, 2019 1:15 PM
>>>
>>>> *To:* Kyzivat, Paul <[email-masked]> 
>>>> <mailto:[email-masked]&gt>;
>>>
>>>> *Cc:* [email-masked] <mailto:[email-masked]>
>>>
>>>> *Subject:* [EXT] Re: [Rum] Real-time text in WebRTC is discussed in 
>>>
>>>> mmusic - a topic closely telated to rum
>>>
>>>>
>>>
>>>> “Line at a time” could be an implementation option.  The protocol 
>>>
>>>> mechanism is all streams head to a mixer and the mixer sends a single 
>>>
>>>> stream to each participant.
>>>
>>>>
>>>
>>>> Gunnar, we agreed we were using 4103 in RUM.
>>>
>>>>
>>>
>>>> Brian
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>      On Aug 29, 2019, at 12:05 PM, Paul Kyzivat 
>>>> <[email-masked] <mailto:[email-masked]>
>>>
>>>>  <mailto:[email-masked]>> wrote:
>>>
>>>>
>>>
>>>>      I question how well RTT text is suited to multiparty conferences.
>>>
>>>>
>>>
>>>>      If you have messages on your screen from multiple parties, and
>>>
>>>>      many of them are updating in real-time, are you going to be able
>>>
>>>>      to perceive what is going on?
>>>
>>>>
>>>
>>>>      And while you can have a column per person for two-party and 
>>>> maybe
>>>
>>>>      3-party conversations, that doesn't scale up. With many parties,
>>>
>>>>      some typing may scroll off the screen before it is complete.
>>>
>>>>
>>>
>>>>      Perhaps for conferences it is better to just use line-at-a-time
>>>
>>>>      chat. If necessary, I presume there could be gateways between RTT
>>>
>>>>      and chat. A RUE could have the capability to negotiate down from
>>>
>>>>      RTT to chat.
>>>
>>>>
>>>
>>>>      Thanks,
>>>
>>>>      Paul
>>>
>>>>
>>>
>>>>      On 8/28/19 2:15 AM, Gunnar Hellström wrote:
>>>
>>>>
>>>
>>>>          Den 2019-08-27 kl. 23:15, skrev Brian Rosen:
>>>
>>>>
>>>
>>>>              At least by centralizing the problem at a “mixer”, Alice
>>>
>>>>              and Bob will see the same thing.
>>>
>>>>
>>>
>>>>              You don’t have the problem in Instant Messaging, because
>>>
>>>>              you can’t backspace or delete a sent message.  Of course
>>>
>>>>              if multiple people are typing simultaneously in such
>>>
>>>>              systems, message order will be confusing in that instant.
>>>
>>>>
>>>
>>>>          Right, it is a similar kind of problem that text appears 
>>>> in an
>>>
>>>>          unexpected order. There is also at least one instant 
>>>> messaging
>>>
>>>>          service that allows modification in already sent message. But
>>>
>>>>          I think it has limitations to only accept that in the last
>>>
>>>>          message sent. it is convenient anyway.
>>>
>>>>
>>>
>>>>
>>>
>>>>              Anyway, we need to specify the mixer for RTT so it
>>>
>>>>              receives each of the RTT streams and produces a single
>>>
>>>>              composite stream for each participant.
>>>
>>>>
>>>
>>>>          Yes, right, and there is an effort in that direction in:
>>>
>>>> http://www.realtimetext.org/sites/default/files/Files_and_Documents/Specifications/multiparty-real-time-text-mixer-2011-04-30.pdf 
>>>>
>>>
>>>>          It is written for conference-unaware user devices.
>>>
>>>>          The goals are specified as follows:
>>>
>>>>          The procedures are intended to make best efforts to present a
>>>
>>>>          multi-party text conversation on a terminal that has no
>>>
>>>>          awareness of multi-party calls. There are some obvious
>>>
>>>>          drawbacks, and a terminal designed with multi-party awareness
>>>
>>>>          will be able to present multi-party call contents in a more
>>>
>>>>          flexible way. Only two parties at a time will be allowed to
>>>
>>>>          display added text in real-time, while the other parties’
>>>
>>>>          produced text will need to be stored in the multi-party 
>>>> server
>>>
>>>>          for a moment awaiting a suitable occasion to be displayed.
>>>
>>>>          There are also some cases of erasure that will not be
>>>
>>>>          performed on the target text but only indicated in another
>>>
>>>>          way. Even with these drawbacks, the procedure provides an
>>>
>>>>          opportunity to display text from more than two parties in a
>>>
>>>>          smooth and readable way.
>>>
>>>> --------------------------------------------------------------------------------------------------- 
>>>>
>>>
>>>>          I see such mixer procedures as a fall-back for cases without
>>>
>>>>          conference awareness, but want to see support for
>>>
>>>>          conference-aware terminals, where text from more than two
>>>
>>>>          parties can be presented in real-time, and the end user or 
>>>> app
>>>
>>>>          can have influence over the presentation style - e.g. select
>>>
>>>>          between the multiple column view and the
>>>
>>>>          one-column-with-labels view. A mixer for that case would only
>>>
>>>>          need to assure that the receiver has the right kind of
>>>
>>>>          multi-party awareness and send RTT text with source
>>>
>>>>          information attached, and let the receiving terminal sort out
>>>
>>>>          the presentation. This is already possible with CSRC and 
>>>> CNAME
>>>
>>>>          when using RTP, but we lose that possibility natively when
>>>
>>>>          using the WebRTC data channel to transport RTT, and would 
>>>> need
>>>
>>>>          to specify a way to include the source also for that case.
>>>
>>>> ------------------------------------------------------
>>>
>>>>          By the way, what is your current view of how to transport RTT
>>>
>>>>          for RUM, now when you say that you will use WebRTC transports
>>>
>>>>          for media?
>>>
>>>>          Regards
>>>
>>>>          Gunnar
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>                  On Aug 27, 2019, at 4:43 PM, Gunnar Hellström
>>>
>>>> <[email-masked] <mailto:[email-masked]>
>>>
>>>> <mailto:[email-masked]><mailto:[email-masked]>> 
>>>>
>>>
>>>>                  wrote:
>>>
>>>>
>>>
>>>>                  Den 2019-08-27 kl. 21:48, skrev Brian Rosen:
>>>
>>>>
>>>
>>>>                      The problem of conference 4103 RTT is high on my
>>>
>>>>                      list of work I need to get done.  So, I’m
>>>
>>>>                      motivated to help out.
>>>
>>>>
>>>
>>>>                  Thanks, great.
>>>
>>>>
>>>
>>>>                      The basic problem is that we’re going to get very
>>>
>>>>                      inconsistent UI doing it that way, because of how
>>>
>>>>                      systems will handle backspace of one party that
>>>
>>>>                      extends beyond responses from other parties:
>>>
>>>>
>>>
>>>>                  (well, for me the currently most basic problem is to
>>>
>>>>                  have a reliable way to append received text to the
>>>
>>>>                  already presented text of the right participant. And
>>>
>>>>                  that is getting worse in WebRTC than it was in RFC
>>>
>>>>                  4103. But we will sort it out.)
>>>
>>>>
>>>
>>>>
>>>
>>>>                      Alice: I waited for you
>>>
>>>>                      Bob: I didn’t see you
>>>
>>>>                      Alice: sorry
>>>
>>>>
>>>
>>>>                      And then Alice types 12 backspaces.
>>>
>>>>
>>>
>>>>                      What should happen?
>>>
>>>>
>>>
>>>>
>>>
>>>>                  You are right that there are a number of ways to
>>>
>>>>                  handle the RTT UI. And just as inconsistencies are
>>>
>>>>                  common with a message oriented UI, where messages 
>>>> show
>>>
>>>>                  up in a confusing order because two users completed
>>>
>>>>                  messages in an unexpected time order, it is possible
>>>
>>>>                  that RTT text gets displayed in a strange order after
>>>
>>>>                  erasure and retyping. It is better for RTT than for
>>>
>>>>                  message oriented presentation, and user get used 
>>>> to it
>>>
>>>>                  in both cases.  With the labelled style in one column
>>>
>>>>                  you have in the example, I would recommend that first
>>>
>>>>                  5 backspaces erase "sorry", next backspace erases the
>>>
>>>>                  line separator, and pulls down "I waited for you" to
>>>
>>>>                  be shown last, as an uncompleted text. Then the 
>>>> next 6
>>>
>>>>                  backspaces erase so that only "I waited f" is
>>>
>>>>                  displayed. When Alice adds text and end with a new
>>>
>>>>                  line, the corrected sentence is allowed to flow up
>>>
>>>>                  when new text is added from any participant.  That
>>>
>>>>                  causes a bit strange order, but it is just as
>>>
>>>>                  manageable as when text in messaging applications
>>>
>>>>                  appear in an unexpected order so that one message
>>>
>>>>                  seems to be a respone on something totally else than
>>>
>>>>                  what was intended.
>>>
>>>>
>>>
>>>>                  A sophisticated UI may mark text that is moved and
>>>
>>>>                  modified.
>>>
>>>>
>>>
>>>>                  We want to keep sentences or at least phrases from
>>>
>>>>                  each participant together in a readable unit. Already
>>>
>>>>                  that causes a design decision on where to place the
>>>
>>>>                  completed chunk of text once the user has completed
>>>
>>>>                  it. The start of the chunk may be older than 
>>>> completed
>>>
>>>>                  text from other participants which would motivate to
>>>
>>>>                  move it up a bit in the presentation. But the end of
>>>
>>>>                  it is at that moment the latest text to present. I
>>>
>>>>                  think it is best to let the finished text be 
>>>> presented
>>>
>>>>                  last on the display, but let others' newer text push
>>>
>>>>                  everything up and be displayed last.
>>>
>>>>
>>>
>>>>
>>>
>>>>                 T.140 has information on how to handle erasure:
>>>
>>>>
>>>
>>>> -------------------From T.140---------------------------
>>>
>>>>
>>>
>>>>                  8.2 Erase last character
>>>
>>>>                  Purpose: Erase the last character sent from the
>>>
>>>>                  display at the receiving end.
>>>
>>>>                  Code: BS: 0008.
>>>
>>>>                  Procedure: On the receiving end: Move the insertion
>>>
>>>>                  point to the last character and erase it.
>>>
>>>>                  Combined characters are erased as a unit, with one BS
>>>
>>>>                  erasing the whole character even if it is
>>>
>>>>                  combined from more than one component.
>>>
>>>>                  Control sequences (like CR LF) are erased in one
>>>
>>>>                  operation.
>>>
>>>>                  NOTE – The same action shall be taken on the local
>>>
>>>>                  display.
>>>
>>>>
>>>
>>>> ------------------------------------------------------------
>>>
>>>>
>>>
>>>>                    /Gunnar
>>>
>>>>
>>>
>>>>
>>>
>>>>
>>>
>>>>                      Brian
>>>
>>>>
>>>
>>>>
>>>
>>>>                          On Aug 27, 2019, at 9:52 AM, Gunnar Hellström
>>>
>>>> <[email-masked] <mailto:[email-masked]>
>>>
>>>> <mailto:[email-masked]><mailto:[email-masked]>> 
>>>>
>>>
>>>>                          wrote:
>>>
>>>>
>>>
>>>>                          Hi,
>>>
>>>>
>>>
>>>>                          A topic is currently discussed in mmusic that
>>>
>>>>                          is closely related to rum. it is WebRTC
>>>
>>>>                          transport of real-time text.
>>>
>>>>
>>>
>>>>                          The draft is
>>>
>>>> draft-holmberg-mmusic-t140-usage-data-channel .
>>>
>>>>
>>>
>>>>                          A good point to start reading could be:
>>>
>>>>
>>>
>>>> https://mailarchive.ietf.org/arch/browse/mmusic/?gbt=1&q=draft-holmberg-mmusic-t140-usage-data-channel 
>>>>
>>>
>>>>
>>>
>>>>                          Please check if the current state of the
>>>
>>>>                          discussion suits rum!
>>>
>>>>
>>>
>>>>                          The only issue that seems to be remaining is
>>>
>>>>                          how to transport RTT data to and from a
>>>
>>>>                          conference server that combines all traffic
>>>
>>>>                          per media in a meeting in one data stream.
>>>
>>>>                          That is not very elegantly specified for RFC
>>>
>>>>                          4103 transport of RTT in RTP either, so we
>>>
>>>>                          might want to do a rapid action together to
>>>
>>>>                          solve the multi-party RTT MCU case in a
>>>
>>>>                          general and consistent way.
>>>
>>>>
>>>
>>>>                          Regards
>>>
>>>>
>>>
>>>>                          Gunnar
>>>
>>>>
>>>
>>>>                          --
>>>
>>>> -----------------------------------------
>>>
>>>>                          Gunnar Hellström
>>>
>>>>                          Omnitor
>>>
>>>> [email-masked] <mailto:[email-masked]>
>>>
>>>> <mailto:[email-masked]>
>>>
>>>>                          +46 708 204 288
>>>
>>>>
>>>
>>>>                  --
>>>
>>>> -----------------------------------------
>>>
>>>>                  Gunnar Hellström
>>>
>>>>                  Omnitor
>>>
>>>> [email-masked] <mailto:[email-masked]>
>>>
>>>> <mailto:[email-masked]><mailto:[email-masked]> 
>>>>
>>>
>>>>                  +46 708 204 288
>>>
>>>>
>>>
>>>>          --
>>>
>>>> -----------------------------------------
>>>
>>>>          Gunnar Hellström
>>>
>>>>          Omnitor
>>>
>>>>         [email-masked] 
>>>> <mailto:[email-masked]> 
>>> <mailto:[email-masked]>
>>>
>>>>          +46 708 204 288
>>>
>>>>
>>>
>>>>
>>>
>>>>      --
>>>
>>>>      Rum mailing list
>>>
>>>>     [email-masked] <mailto:[email-masked]> <mailto:[email-masked]>
>>>
>>>> https://www.ietf.org/mailman/listinfo/rum
>>>
>>>>
>>>
>>>>
>>>
>>> -- 
>>>
>>> -----------------------------------------
>>>
>>> Gunnar Hellström
>>>
>>> Omnitor
>>>
>>> [email-masked] <mailto:[email-masked]>
>>>
>>> +46 708 204 288
>>>
>>>   * [Rum] Real-time text in WebRTC is discussed in ...
>>> <https://mailarchive.ietf.org/arch/msg/rum/hzX30Lo4DY4J3RGbtgxoC5wpE1s>  Gunnar 
>>>
>>>     Hellström
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/Lk90w0qStmiAmHoIw_mWFTOE3pY>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/opt7uhe3jcCAGEXipXCefVN97PE>  Gunnar 
>>>
>>>     Hellström
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/uF83bn9GmEkR2FzE4rvH-DiJn1I>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/6402q9HzKBNqvkwKajSlo4EYmMU>  Gunnar 
>>>
>>>     Hellström
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/YG02TyKon1tRtctZ1Bwe0BnI7Cw>  Paul 
>>>
>>>     Kyzivat
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/UDz0JCzyqQ0ut8RJzWF54KjMUXY>  Malloy, 
>>>
>>>     Jim
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/WWekfAbXC81GFmOdUld1YOvV450>  Paul 
>>>
>>>     Kyzivat
>>>   * Re: [Rum] Real-time text in WebRTC is discussed...
>>> <https://mailarchive.ietf.org/arch/msg/rum/8N5sWJvqB-J4XgC5xqMCV2nL0gc>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/HoD8Ni1r0f8S11H2K4hp6Ss5NB0>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/eI-dX4HrMUur324GEt-CFmpv4mY>  Paul 
>>>
>>>     Kyzivat
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/AaoKog-HhtBUS_a-BKrB5BpwiF4>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/1JojOtitB-FlcmCHQd9LMl4PWM4>  Paul 
>>>
>>>     Kyzivat
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/2NDD2kMSGngnh6GyoLD9Th_UYrM>  Brian 
>>>
>>>     Rosen
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/UZDyKrfOw-bjkbqjN8E3MHXrseo>  Malloy, 
>>>
>>>     Jim
>>>   * Re: [Rum] [EXT] Re: Real-time text in WebRTC is...
>>> <https://mailarchive.ietf.org/arch/msg/rum/avASfCcxCAY18gOFsTGE3P5oTOs>  Gunnar 
>>>
>>>     Hellström
>>>
>>> -- 
>>>
>>> *John Martin*
>>>
>>> Email: [email-masked] <mailto:[email-masked]>
>>>
>>>
>>
-- 
-----------------------------------------
Gunnar Hellström
Omnitor
[email-masked]
+46 708 204 288

---

From: Brian Rosen <[email-masked]>
Date: Mon, 2 Sep 2019 10:29:55 -0400

I think it’s of marginal utility.  The reason you need muting for audio is that you can’t process multiple simultaneous speakers.  In RTT, it’s a lot easier.  It makes more sense in conference control (not from the endpoint, controlling what the mixer does, a la xcon).  OTOH, it ought to be the same as audio and video.

Brian

> On Sep 1, 2019, at 5:26 PM, Gunnar Hellström <[email-masked]> wrote:
> 
> I would like a comment from RUM if it is desirable to have support for the direction attributes : sendrecv, sendonly, recvonly, inactive,  for RTT streams?
> 
> It might be for control of muting etc in multi-party situations, but of course also in other situations.
> 
> Regards
> 
> Gunnar
> 
>>> Thanks for joining in! I don't have much knowledge of (and no experience using) RTT so I will mostly watch from the sidelines as long as others join in.
>>> 
>>>     Thanks,
>>>     Paul
>>> 
>>> (BTW, I don't know how you formatted your message, but in my mail reader (Thunderbird) none of your lines wrapped, requiring horizontal scrolling to read them.)
>>> 
>>>> RTT in General
>>>> 
>>>> --------------
>>>> 
>>>> ISTM there are a few options for RTT in RUM, assuming the WebRTC media framework is adopted (as seems likely):
>>>> 
>>>> 1.Use existing browser support for SCTP as recommended by the some of the WebRTC IETF WG participants. This was discussed as part of the IVC WG and would require and SCTP data channel to be negotiated and to encapsulate the RTP (4103) which then in turn encapsulate T.140 RTT. This solution would require gateways in the VRS provider networks, RTP/RTCP userspace (Javascript) layers in the browser and a T.140 RTT layer/UI also written in userspace in the browser.
>> 
>> The current work discussed in mmusic  about draft-holmberg-mmusic-t140-usage-data-channel  specifies T.140 real-time text transported directly in a "reliable" WebRTC data channel, identified with subprotocol "t140". Would that not be suitable for a WebRTC based RUM? The data payload will be the same as carried in RFC 4103 if used without redundancy. You mention that embedded RTP with RFC 4103 would be used as data also in the data channel. That sounds complicated. What would you gain?
>> 
>>>> 2.Wait for WebRTC to support lower level data channels. This work is being discussed in the WebRTC community, but might be a case of “not holding our breath”. A lower level data channel would allow RTP/RTCP then T.140 to be negotiated in the SDP without the need for SCTP and associated gateway needs.
>> Since the RTCWEB WG is closed now in IETF, this sounds as a 7 year uncertain plan.
>>>> 
>>>> 
>>>> 3.As a stop-gap and simplification, support SIP MESSAGE as already provided for by the browsers and many endpoints, including WebRTC endpoints.
>> SCTP data channels are supported as well, and you can just agree to start using it with a private subprotocol temporarily among RUMs until there is a standard. So why use SIP MESSAGE?
>>>> 
>>>> 
>>>> RTT in a conference is not currently defined and does not work using currently defined standards (I must admit to being being on mmusic so perhaps there are solutions being recommended there). Discussions on this have been ongoing for nearly 10 years (we didn’t solve it then Gunnar and I think the problems are still the same). For RTT to be useful in a conference there needs to be a few problems solved.
>> 
>> If you do it with multiple streams sent to the clients, then there are no problems. Therefore there was hesitation when we discussed to bring up the topic in IETF.
>> 
>> For mixing into one stream, there are problems. You can do it with normal RTP and conferencing specifications but get some limitations. You can select between a presentation planned by the mixer, or use CSRC in RTP to indicate the source of primary data in each packet and let the client plan the presentation. Both work and do not really need any further standardisation, but would benefit from good best practice specifications or even a standard for solving some problems.
>> 
>> Planning and formatting the presentation in the mixer has the limitation that only the local user and one at a time of the remote users text can be presented in real-time. Text from the others need to be buffered until the currently presenting remote participant makes something so that giving turn to next in queue is suitable. Switching turn can be made automatically, based on completed phrase or sentence or message or inactivity or allowed maximum time by the presenting party. Identification about who is sending which text needs to be formatted by the mixer and is likely best in form of an IRC style label when the source is switched. This works for a call where users are reasonably well-behaved in taking turns. There are cases where the presentation in one column with sources in labels is not what the user wants.
>> 
>> Using just one stream, and mark by CSRC who contributed to primary data in each packet works as long as no packet was lost. The receiver can make a better adapted presentation for presentation preferences by the user. And text can be presented in real-time from more than two users at the same time.
>> 
>> But in order to use the redundant RFC 4103 data to recover text in case of packet loss, then also the redundant data needs to be marked with source. And for that, a convention is needed and standardised. A couple of variants are possible with different pros and cons. Losing so many packets that a missing text marker needs to be inserted also has its problems that needs a specified solution.
>> 
>>>> 
>>>> 1.Multi-stream support: Brian, whilst you’re right that most conference systems are point-to-point for signalling they are more typically point-to-multi-point for media. The conference bridge usually handles the multiplexing of the streams, but they are usually now multiple streams. RTT conferencing needs to have the same support. We should assume conferencing is supported by way of SFU not MCU – that’s how WebRTC handles conferencing and we should assume the same for RTT.
>> Yes, that is now the main path for draft-holmberg-mmusic-t140-usage-data-channel, and I think that is good. That is also possible without any further standardization for RTP based RTT.
>>>> 
>>>> 2.Participant identification: streams from the conference bridge (MCU/SFU) must be tagged with the sender so the UI can signal to the user who sent that text (and reconstruct the stream correctly).
>> Yes. Use CSRC / CNAME for RTP and  stream ID / label / session information for WebRTC data channel.
>>>> 
>>>> 3.UI adaptions: Modern WebRTC based multi-party conferences have adapted UIs to display multiple video streams. UI’s are adapted to show the video participants however the user wants them to be displayed (single full screen, tiled, voice activated etc). We should expect no less from RTT conference enabled endpoints. It should be a user decision (if their UI supports it) as to whether they see line-by-line or RTT rendering of the text conversation.
>> Yes. If you want a one column presentation, a reasonable presentation is to automatically assign one remote party at a time to be presented in real-time and present the others phrase by phrase, and automatically switch who is presented in real-time based on suitable criteria.
>>>> 
>>>> 4.Control of how many simultaneous participants can text and how that is presented to the user needs to be explored. Multiple insertion points in the UI is one approach (though rapidly breaks down as the number of users increases). Line-by-line is another. Facilitated GA is another (the Go Ahead concept has been used for a long time in text conversations and may be appropriate in large RTT conferences to control overload).
>>>> 
>>>> And a final FWIW: I think the utility of RTT in a conference quickly breaks down as the number of participants increases. It is completely unacceptable to have a single RTT stream presented to the user and an unmodified UI. You might as well revert to SIP MESSAGE and something like XMPP, and that could also be an option for either the short term or for endpoints that do not support multi-stream RTT.
>> 
>> In some conferences it might be desirable with one dominating RTT source: a written interpretation of what is said, and then occasionally another participant gets the floor and types something . It is very valuable if that mode of conferencing is available without hazzle in general multi-party conference systems. The utility does not break down in that situation.
>> 
>>>> audio and video. Can you make RTP-based RTT work in that environment and
>>>> 
>>>> Well, the control and coding must be made so that it is possible to make
>>>> 
>>>>> *From:* Rum <[email-masked]> <mailto:[email-masked]&gt>; *On Behalf Of * 
>>>> Brian Rosen
>>>> 
>>>>> *To:* Kyzivat, Paul <[email-masked]> <mailto:[email-masked]&gt>;
>>>> 
>>>>>      On Aug 29, 2019, at 12:05 PM, Paul Kyzivat <[email-masked] <mailto:[email-masked]>
>>>> 
>>>>>      And while you can have a column per person for two-party and maybe
>>>> 
>>>>>          Right, it is a similar kind of problem that text appears in an
>>>> 
>>>>>          unexpected order. There is also at least one instant messaging
>>>> 
>>>>>          produced text will need to be stored in the multi-party server
>>>> 
>>>>>          parties can be presented in real-time, and the end user or app
>>>> 
>>>>>          the presentation. This is already possible with CSRC and CNAME
>>>> 
>>>>>          using the WebRTC data channel to transport RTT, and would need
>>>> 
>>>>>                  common with a message oriented UI, where messages show
>>>> 
>>>>>                  message oriented presentation, and user get used to it
>>>> 
>>>>>                  be shown last, as an uncompleted text. Then the next 6
>>>> 
>>>>>                  it. The start of the chunk may be older than completed
>>>> 
>>>>>                  think it is best to let the finished text be presented
>>>> 
> [email-masked]

---

From: Gunnar Hellström <[email-masked]>
Date: Mon, 2 Sep 2019 17:40:14 +0200

> I think it’s of marginal utility.  The reason you need muting for audio is that you can’t process multiple simultaneous speakers.  In RTT, it’s a lot easier.  It makes more sense in conference cont…

Thanks, I think we have suitable wording in 
draft-holmberg-mmusic-t140-usage-data-channel for the direction 
attributes now. At the moment not as session level attributes though, 
but that might be less important than having them on the channel level.

/Gunnar