Paul,
I have no doubt that the proper procedures were followed and am not
questioning that aspect. I was active in one of SG16's predecessor
committees for ten years and at various times an editor and rapporteur, and
hence am sympathetic to the need to be fair, reasonable and "procedural".
We have been studying the nature and source of IP network impairments and
their effects on the performance of VoIP for over three years - basically
that is all we do :-). We have done extensive analysis of the full range of
G series CODECS, the various GSM codecs and a number of proprietary
algorithms under a range of packet loss conditions. We've measured and
modeled the behavior of fixed and adaptive jitter buffers of various types.
We built a VoIP service modeling environment using NS2 with MMPP traffic
models and used this for analyzing the precise behavior of individual calls
under known traffic conditions or network faults. We have analyzed RTP
traces from a wide variety of sources..... Our algorithms are in use by a
number of major service providers - due in part to the way in which we view
IP network behavior as time varying/ transient/ bursty and hence produce
metrics in a way that reflects this.
The result of this extensive research activity is (I hope) that we have a
deep technical understanding coupled with field experience on the merits of
different approaches. We have also discussed this topic with others working
in the field - in both industry and academic research groups - and find that
our observations match very well with those of other groups that have
studies this issue deeply.
As we discuss the issue of VoIP service quality monitoring with equipment
vendors, enterprise network managers and service providers we often get
asked the question "why aren't RTCP statistics good enough" and also see
products on the market that use RTCP statistics to estimate VoIP service
quality (and even try to estimate discard rates from jitter buffers!!).
Obviously, from my comments on RFC1889 yesterday, we think that RTCP
statistics are only adequate at a gross level - i.e. if average packet loss
is 30% call quality must be bad but you could not determine if a call with
an average loss rate of 2% was better or worse than one of 3%.
I am sure that if anyone else were in our position then they would similarly
feel very concerned at the idea that it was acceptable to incorporate the
RFC1889 metrics into a new ITU Recommendation that was positioned as a VoIP
QoS monitoring standard. Many manufacturers and service providers do not
have the technical expertize to evaluate the merits and detailed content of
international standards, particularly in a complex area such as this, and
tend to assume that the content is ok "because it is in a standard". This
would naturally tend to proliferate the use of these metrics as an
acceptable approach - even though we know that they are inadequate.
I hasten to add that I am not criticizing you, the Rapporteur's Group, the
SG16 management or the ITU .. this (in my opinion) happens to be one of the
complex issues in which what appears to be the correct and proper course of
action can lead to a problematic result.
Regards
Alan Clark
-----Original Message-----
From: Paul E. Jones [mailto:paulej@packetizer.com]
Sent: Thursday, November 07, 2002 1:05 AM
To: alan(a)telchemy.com
Cc: ITU-SG16(a)echo.jf.INTEL.COM
Subject: Re: Use of RTCP statistics for estimating QoS of VoIP
Alan,
This document (now H.460.9) first appeared about a year ago. After the
author presented the paper, it was agreed without debate to create a new
work item for it. At this last meeting, the author presented the paper, as
scheduled, for approval. During interim meetings, there were no
contributions against the document, which is the accepted procedure for
raising technical issues and trying to make changes to progressing work
items.
During the last meeting, Mike Buckley did express concern over this
document. I think Mr. Buckley's concerns were all valid and he pointed us
to this Internet Draft. However, I had a decision to make: to hold progress
on this document until the IETF produces an RFC or to allow the document to
go forward as scheduled and as desired by the contributor.
Given that this contributing company had made the document available about a
year ago and received no formal contributions against it, it really was not
my place to hold up progress at the last moment. While I certainly had the
wherewithal to put the document on hold, I just didn't think that is the
right thing to do and it certainly was not the desire of the contributor.
With that said, though, we did make changes at the meeting to the document
to prepare for this forthcoming RFC. It was agreed that we will create
either an Annex or Amendment to add the necessary fields to the existing
data structures so that the additional information can be reported once the
RFC is available.
Unfortunately, Mr. Buckley was not available during the second week at the
meeting to discuss some of those decisions. Even some of my communication
with him via e-mail was unsuccessful due to bounced messages. We hoped that
the decisions we made would be acceptable to him. While I definitely value
the input he provides and I hold his opinions in high regard, I felt like it
was better to publish this document now with the aforementioned changes and
plans, so as to meet the wishes of the contributor and try to address the
concerns of Mr. Buckley at the same time. After all, there was no guarantee
that the RFC would be published between now and the next SG16 meeting in May
2003 and we certainly cannot wait indefinitely on the IETF to complete work.
Paul
----- Original Message -----
From: "Alan Clark" <alan(a)telchemy.com>
To: <paulej(a)packetizer.com>
Cc: <ITU-SG16(a)echo.jf.INTEL.COM>
Sent: Wednesday, November 06, 2002 10:49 AM
Subject: Use of RTCP statistics for estimating QoS of VoIP
>
> Paul
>
> I am responding to the recent SG16 email that encouraged direct input from
> those not participating directly in the work of the Study Group.
>
> One particular recent issue relates to the recent addition of QoS
reporting
> based on RTCP statistics. We have spent the last three years analyzing
and
> modeling the effects of IP network impairments on VoIP and would like to
> share our experience on this issue.
>
> RTCP statistics were intended to provide a "rough" real time feedback
> between endpoints to give some sense of network quality and facilitate
> potential adaptation. IP network impairments are transient in nature and
> the "averaged" metrics that RFC1889 produces are as a result very
> misleading. The addition of QoS reporting based on RFC1889 metrics is, in
> our opinion, a backward step as it infers that these statistics are useful
> for VoIP quality estimation whereas they are in fact almost meaningless.
>
> (i) Packet Loss - counts
> RFC1889 counts of loss are not compensated for duplicate packets, hence it
> is possible to have incorrect and even negative (!!) loss counts.
>
> (ii) Packet Loss- distribution
> Voice quality is very sensitive to the distribution of packet loss - if 2%
> loss occurs but the losses are spread out evenly then with PLC quality
would
> be affected only marginally, if 1% loss occurs but losses occur during
short
> intervals (typically the case) then quality can be affected quite
> significantly.
> RFC1889 does not consider the distribution of lost packets and hence
packet
> loss numbers can only grossly be equated to call quality (i.e. loss rate
of
> 20% is bad but you couldn't say for sure if 2% was bad or not)
>
> (iii) Jitter - metric
> The average packet-to-packet delay variation used in RFC1889 does not
> properly reflect the effects of congestion related delay changes. For
> example
> (a) if congesion occurs and a queue starts to fill then delay increases
> however the packet-to-packet delay change may be small however the
increase
> in delay may cause the jitter buffer to resynchronize at some stage.
> (b) LAN congestion can cause short term spikes in delay, and hence may
lead
> to packet discards. The average packet-to-packet delay variation will not
> reflect this due to the smoothing effect of the averaging algorithm.
>
> (iv) Jitter - reporting interval
> As jitter is reported as a running average with a scaling factor of 16, an
> RTCP report only describes the jitter level during the 200mS or so prior
to
> the report being generated and says nothing about the jitter level during
> the much larger period between reports.
>
> We have spoken to, and worked with, people in the industry that have been
> looking at this issue in detail and have found general agreement with the
> comments above. Unfortunately, there are also a fair number of people
that
> blindly accept that something must be "ok" because it is "in a standard"
..
> this obviously raised the concern that incorporating these metrics into
the
> H series of standards further proliferates this viewpoint.
>
> Within the IETF AVT group this issue was discussed almost a year ago, and
> the group agreed to develop a set of metrics that were more accurate and
> appropriate for VoIP QoS reporting purposes. These include measures of
> packet loss, packets discarded due to jitter, length and density of bursts
> and gaps (of combined loss and discard), delay, call quality metrics...
> This is contained in the following:-
>
> draft-ietf-avt-rtcp-report-extns-01.txt
>
> I would very much appreciate understanding the technical rationale behind
> incorporating RFC1889 based metrics into VoIP QoS reports, given the
> comments above.
>
> Regards
>
> Alan Clark
> Telchemy
>
>
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For help on this mail list, send "HELP ITU-SG16" in a message to
listserv(a)lists.intel.com
