This document has been reviewed as part of the transport area review team's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors and WG to allow them to address any issues raised and also to the IETF
discussion list for information.

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC
tsv-art@ietf.org if you reply to or forward this review.

Dear authors,

let me make a long story short: this more than just a tsv-art review as I am sorry to say that I have more fundamental concerns with the document overall, its contents, its readability, technical detail and precision, and and how useful it is on its own.

Addressing these concerns will take more than a few edits but I actually ask you to go back to square #1 and do a more or less complete rewrite.  Hence, I am not losing myself in details on a given page but provide broader feedback.

Let me also say that I appreciate the underlying idea (even though I am not sure we ned yet another Quality of Something acronym, least with an underspecified meaning).  I also surely appreciate looking at more than just throughput, so the basic inputs towards QoS, QoE, etc. seem about right.

First of all: the document is way to verbose: there is lots of text that doesn‘t lead in a stringent fashion to anything but it doesn‘t provide context or background either. This holds for sections 1-4. The reader reads sentences and gets a vague idea what you may be talking about but quite a bit of guesswork what is important and what isn‘t.  There is also lots of repetition across the sections but those don‘t make it clearer.  In essence, what the document currently does is _talk about things_ (which you can understand if you also know those) but it doesn‘t _explain_ or _describe_ things.

Please have a concise introduction in section 1.

Please provide a concrete description of TR-452.1 and whatever else is needed top put the document properly into context.  This could be a kind of background section 2.  And please make a clear case why one (the IETF, operators, etc.) should care.

You make extensive reference to a framework but, besides a few equations, it is not specified what this framework exactly comprises.

You make reference to „requirements“ in section 4.1 but what you write there aren‘t requirements, or at least not written in a requirements style.

Listing arbitrary techniques how to measure something may be useful for illustration purposes. It may be helpful to clearly separate those from the „requirements“ and follow a similar structure for all subsections.

In section 5, you discuss metrics and composability: one can argue whether metrics are composable or not depending on what you provide as input.  If you had distributions available, the outcome would be different. But it seems that the choice you make is arbitrary (some percentiles or so), without much discussion of tradeoffs. Is this the desirable outcome or is this just what you implemented?

The first sentence of section 5.1 (and not just this statement) wants to be backed up by citations.

I would argue against 5.1.1 — the statement that there is no correlation between throughput and application performance also deserves citations.  There is clearly a correlation if throughput is too low.  I would also contend that throughput cannot be composed unless you can explain to me why not and under which assumptions.

You could certainly compose distributions.

How do you measure some metrics (such as RPM) on individual network segments?

Section 6 needs to account for more details — along the lines the limitations outlined in section 9.  I would argue that these limitations should be dealt with first properly, and then we can move ahead towards an RFC. This is not a scientific paper where we can defer the complicated things to future work. 

Section 6.1 creates an unnecessary issue with elimiating distribution reporting for perceived complexity. This may hold for some networks and devices but may be perfectly fine to achieve for others.  You could have a „light“ reporting mode if you really want to do the simple version, too. This has profound impact on your above composability (which you talk about before providing the reasons why here).

How does your measurement path cope with load balancing?

You will need to provide details on how to specify all metadata exactly.

Section 6.2: Network properties are measured (as you describe) in multiple dimensions. How can you assume an ordinality, let alone a linearity between „too bad“ and „optimal“?

What is section 6.3 supposed to convey?

Section 7 lacks specifying the framework and the specific details. This is left rather vague.

One bullet states:

„The network performance falls between optimal and unacceptable. In this case, a continuous QoO score between 0% and 100% is computed by taking the worst score derived from latency and packet loss.“
You will to provide evidence for this being sensible given the admittedly quite diverse application requirements.

Overall, the document may also benefit from some illustrations.

I also substantial technical concerns:

1. There is no evidence provided that the linear distance measure is actually the right thing to do. Many interpolations are conceivable and, given that you do not specify a specific metric, one should also appreciate that different metrics might need different interpolations.

2. How does your reporting work with continuous measurements? Which data points do you include in distributions or percentiles reported?  Last hour, day, week, month, year, decade?  

3. The limitations of section 9 really need to be addressed first for this to be practically useful.

Best,
Jörg