Alan,

I have just had a look at the final version sof H.460.9 and I could not find any text that explains the real value of the QoS information that can be transported (not including future extensions of course). This is a real pity, because I do agree that many implementors will not be well informed enough to determine the value of the information from the standard RTCP packets.

Regards,

Frank

Alan Clark alan@TELCHEMY.COM Sent by: Mailing list for parties associated with ITU-T Study Group 16 ITU-SG16@echo.jf.INTEL.COM 2002-11-08 18:29 Please respond to alan

To: ITU-SG16@echo.jf.INTEL.COM cc: (bcc: Frank Derks/HVS/BE/PHILIPS) Subject: Re: Use of RTCP statistics for estimating QoS of VoIP Classification:

Frank

I did suggest (via Mike Buckley) that the following be added to H.460.9 however don't know if it was actually included.

" It should be noted that RTCP reports were not originally intended to provide accurate QoS metrics but for informative exchanges between endpoints to support possible dynamic configuration. According to RFC1889 (a) received packet count may include duplicate packets and hence it is possible to report negative packet loss (b) reported mean packet-to-packet delay variation (jitter) is averaged over the last 16 received packets and hence provides information only on the 160-480mS period immediately preceding transmission of the RTCP report."

I do agree with you that potential implementors may well feel uncomfortable that the Recommendation in its initial form contains information that has little or no value. I am even more concerned that less well informed implementors may accept what is in the Recommendation as being "sufficient" and hence attempt to manage VoIP services with metrics that can be quite misleading.

Regards

Alan

-----Original Message----- From: frank.derks@philips.com [mailto:frank.derks@philips.com] Sent: Friday, November 08, 2002 2:46 AM To: alan@TELCHEMY.COM Cc: ITU-SG16@echo.jf.INTEL.COM Subject: Re: Use of RTCP statistics for estimating QoS of VoIP

Alan,

I certainly agree that, if the main body of the Recommendation covers metrics that are of little practical use of the recipient of these metrics, the Recommendation can be misleading. Only with the proper background information will the reader be able to distinguish the real value in the particular metrics.

I feel somewhat uncomfortable when the main body of a Recommendation provides me with something that is of little value, but which can be extended to include more useful information.

A possible way "out of this", is to include some text that clearly explains the situation and that reflects the concerns that were raised and agreed upon.

Regards,

Frank

Bob

I appreciate that a mechanism was provided to permit extension of H.460.9 and that it was agreed to incorporate the RTCP Extensions, once standardized. This does make a lot of sense and would definitely accomodate the new metrics when available.

The reason behind my email yesterday was that we often find that equipment manufacturers, service providers and network managers don't understand this area as well as those who are actively conducting R&D in this area, and hence in this case they may assume that RFC1889 metrics are "good" for monitoring VoIP QoS since they are contained in H.460.9. This is already a continual re-education process with the metrics in RFC1889 .. their inclusion in H.460.9 will make the task even more difficult.

People within the industry have great respect for the ITU, understand that there is an extensive process of technical discussion and review that is part of the process of developing ITU standards, and hence place considerable faith in the technical content of ITU Recommendations. In this specific case H.460.9 contains metrics which are known to be inadequate for the purpose of measuring VoIP QoS and hence there is a risk of deployment of an incorrect or inappropriate element due to its incorporation in an ITU Recommendation.

I am not trying to be "difficult" but tried to input essentially the same viewpoints prior to the SG16 meeting and am genuinely concerned over the outcome.

Regards

Alan Clark

-----Original Message----- From: Mailing list for parties associated with ITU-T Study Group 16 [mailto:ITU-SG16@echo.jf.INTEL.COM]On Behalf Of Robert R. Gilman Sent: Wednesday, November 06, 2002 8:13 PM To: ITU-SG16@echo.jf.INTEL.COM Subject: Re: Use of RTCP statistics for estimating QoS of VoIP

Alan- Mike Buckley brought up the same questions you raise at the last SG16 meeting. I don't believe anyone objects to your arguments, and, in fact, the resulting work, H.460.9, was modified to permit extension to other measures and the exclusion of the current, IETF-standardized, measures. Once the IETF draft you reference becomes a standard, it will be easy to refer to the resulting RFC and to incorporate the resulting measures as appropriate. In the meantime, you have the option to use the existing measures and, through the definition of non-standard elements, any other measures you wish to implement on a permanent or trial basis. It seemed better to set the framework now, so that implementation and use can begin, and to do so in such a way that future improvements may be incorporated quickly and easily. Does this make sense? -Bob ---------------------------------------------------- Bob Gilman rrg@avaya.com +1 303 538 3868

Alan Clark wrote:

Paul

I am responding to the recent SG16 email that encouraged direct input

from

those not participating directly in the work of the Study Group.

One particular recent issue relates to the recent addition of QoS

reporting

based on RTCP statistics. We have spent the last three years analyzing

and

modeling the effects of IP network impairments on VoIP and would like to share our experience on this issue.

RTCP statistics were intended to provide a "rough" real time feedback between endpoints to give some sense of network quality and facilitate potential adaptation. IP network impairments are transient in nature

and

the "averaged" metrics that RFC1889 produces are as a result very misleading. The addition of QoS reporting based on RFC1889 metrics is,

in

our opinion, a backward step as it infers that these statistics are

useful

for VoIP quality estimation whereas they are in fact almost meaningless.

(i) Packet Loss - counts RFC1889 counts of loss are not compensated for duplicate packets, hence

it

is possible to have incorrect and even negative (!!) loss counts.

(ii) Packet Loss- distribution Voice quality is very sensitive to the distribution of packet loss - if

2%

loss occurs but the losses are spread out evenly then with PLC quality

would

be affected only marginally, if 1% loss occurs but losses occur during

short

intervals (typically the case) then quality can be affected quite significantly. RFC1889 does not consider the distribution of lost packets and hence

packet

loss numbers can only grossly be equated to call quality (i.e. loss rate

of

20% is bad but you couldn't say for sure if 2% was bad or not)

(iii) Jitter - metric The average packet-to-packet delay variation used in RFC1889 does not properly reflect the effects of congestion related delay changes. For example (a) if congesion occurs and a queue starts to fill then delay increases however the packet-to-packet delay change may be small however the

increase

in delay may cause the jitter buffer to resynchronize at some stage. (b) LAN congestion can cause short term spikes in delay, and hence may

lead

to packet discards. The average packet-to-packet delay variation will

not

reflect this due to the smoothing effect of the averaging algorithm.

(iv) Jitter - reporting interval As jitter is reported as a running average with a scaling factor of 16,

an

RTCP report only describes the jitter level during the 200mS or so prior

to

the report being generated and says nothing about the jitter level

during

the much larger period between reports.

We have spoken to, and worked with, people in the industry that have

been

looking at this issue in detail and have found general agreement with

the

comments above. Unfortunately, there are also a fair number of people

that

blindly accept that something must be "ok" because it is "in a standard"

..

this obviously raised the concern that incorporating these metrics into

the

H series of standards further proliferates this viewpoint.

Within the IETF AVT group this issue was discussed almost a year ago,

and

the group agreed to develop a set of metrics that were more accurate and appropriate for VoIP QoS reporting purposes. These include measures of packet loss, packets discarded due to jitter, length and density of

bursts

and gaps (of combined loss and discard), delay, call quality metrics... This is contained in the following:-

draft-ietf-avt-rtcp-report-extns-01.txt

I would very much appreciate understanding the technical rationale

behind

incorporating RFC1889 based metrics into VoIP QoS reports, given the comments above.

Regards

Alan Clark Telchemy

For help on this mail list, send "HELP ITU-SG16" in a message to
listserv@lists.intel.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For help on this mail list, send "HELP ITU-SG16" in a message to listserv@lists.intel.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For help on this mail list, send "HELP ITU-SG16" in a message to listserv@lists.intel.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For help on this mail list, send "HELP ITU-SG16" in a message to listserv@lists.intel.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For help on this mail list, send "HELP ITU-SG16" in a message to listserv@lists.intel.com