Robert
Firstly I would like to thankyou for the fantastic work you did on upgrading
the initial Video Plugin work on the openH323_plugins branch.
On to this discussion. I asked a specific question on the removal of the
frame size constraints in the encoder of the plugin codec. Now I don't want
to break anything so from your indepth response about YUV files and SIP
exchange example. I will assume its important so I will just leave it as it
is for Opal. :-)
Regards
Simon
-----Original Message-----
From: h323plus-bounces(a)lists.packetizer.com
[mailto:h323plus-bounces@lists.packetizer.com]On Behalf Of Robert
Jongbloed
Sent: Wednesday, 7 November 2007 2:15 PM
To: Opalvoip-devel(a)lists.sourceforge.net
Cc: 'H323plus'
Subject: Re: [h323plus] [Opalvoip-devel] Custom Video Frame Size
> -----Original Message-----
> From: Simon Horne [mailto:s.horne@packetizer.com]
....
> Getting back to the initial question. I want to move forward with
> H.239
> support in h323plus so can I remove the fixed frame size constraints
> from
> the video plugins so the project can move forward or if that's not
> recommended then, as I don't want to have different versions of the
> video
> plugins that break interoperability, can I put in a compiler
> directive to
> get us out of a pickle? Once these opal architectural glitches are
> resolved
> then the directive can be removed.
I am confused, the underlying system in OPAL has never really had any "fixed
size" constraints. There are two OpalMediaOptions for width and height and
they can be any value. Now come CODECS can only do fixed sizes, eg H.261,
and that was part of the complexity Matthias and I were struggling with.
> I really am confused on the codec issues, and the discrete video
> sizes with
> H.261/H.263 and the generic capabilities etc. The way this is done in
> H323plus is to detect the capabilities of the video device at
> application
> startup via the changes I made in the ptlib videodevice factory which
> allows
> the device capability list to be exposed without instantaneousing the
> device. You use the device capabilities list to determine the maximum
> frame
> size available for the device so in this way you can detect and
> support HD
> webcams etc. There is a H323Endpoint function that then goes through
> and
> removes all the capabilities unsupported for that particular webcam.
> Easy!
If I am reading this right, you are using the capabilities of the camera,
which is used for transmit channels, to determine the H.323 capabilities
which controls the RECEIVE channels. Surely the capability should use the
VideoOutputDevice? Or is this for transmitVideo capabilities only?
Also, but what happens if you have a camera that only reports being able to
do 320x240? Not sure this happens much anymore, but older cameras certainly
used to. Again, if I am reading what you said right, you would not get H.261
or H.263 at all as they can't do that resolution.
> On the OpenVideoChannel function callback the user can then set the
> frame
> size and fps on the wire. This sets the header height/width fields of
> the
> YUV420 frame which then goes back into the plugin codec to resize the
> codec.
> This is how it used to work in OpenH323 and it works just fine. The
> problem
> you refer to is, I guess, an open Opal issue perhaps?.
Given recent events I would be VERY careful about gibes like this.
Here is the problem as I see it, library neutral ...
Leaving out for the moment the added complexity of requiring symmetric
codecs, the receive and transmit video streams are completely independent.
Let's start with the receiver; there are three entities at work:
Video output device capabilities
User/Application preferences
Codec fundamentals
As a rule most output devices can do any resolution/frame rate, however it
is possible with the YUVFile driver to indicate that it MUST be a size, e.g.
using the filename "fred_qcif.yuv".
The User/Application may also have restrictions, most common is for PDA's
where the screen size is such that you want to prevent 4CIF etc.
The Codec fundamentals are the most complicated as they may just be a
maximum (via profile/level for H.264/MPEG4 and CustomPictureFormat for
H.263+) or a set of discrete values as required by H.261 and H.263.
The presented capabilities sent to the remote must be derived from the
above. I am not presenting a solution yet, just trying to state the problem
in as complete a form as possible.
For the transmit side we have four entities:
Video input device capabilities
User/Application preferences
Codec fundamentals
Remote capabilities
So, when selecting our specific video parameters we want to try and get as
close to the user preference as possible given the constraints indicated by
the other three entities.
One more random point: the Codec fundamentals are only known by the plug-in.
So, the result of a long discussion with Matthias (and others, but Matthias
was the main person) we came up with a first cut solution. To simplify
matters an assumption that the video devices can be told to go to any
resolution is made. As we have the PColourConverter functionality which
(mostly) can do scale/crop as it converts, I think that is fair.
Then we introduce the concept of "normalised" OpalMediaOptions and "custom"
OpalMediaOptions. The normalised options are things like min/max
width/height and the custom options are things like profile/level, or "QCIF
MPI".
Then two functions are to be added to the plug in to set the options in an
instance from normalised to custom, and back from custom to normalised
options. In the process the plug-in can also apply any other rules it might
have, such as discrete sizes.
And here is the sequence that I am stealing from the other thread describing
how all the OpalMediaOptions get tweaked as they pass through the system:
OK, nothing better than a concrete example:
Media format on start up from a H.263 plug-in:
Max Frame Width = 1408
Max Frame Height = 1152
Min Frame Width = 144
Min Frame Height = 96
Frame Width = 352
Frame Height = 288
Max Bit Rate = 384000
Target Bit Rate = 384000
Frame Time = 3000
SQCIF MPI = 1
QCIF MPI = 1
CIF MPI = 2
4CIF MPI = 3
16CIF MPI = 4
Annex D = 1
Then the user alters the following:
Max Frame Width = 640
Max Frame Height = 480
Frame Width = 320
Frame Height = 240
Max Bit Rate = 128000
Frame Time = 6000
Then just before making a call the above, after user adjustment, is sent to
the plug-in "from_normalised_options" function, the plug-in returns:
Max Frame Width = 352
Max Frame Height = 288
Min Frame Width = 144
Min Frame Height = 96
Frame Width = 176
Frame Height = 144
Max Bit Rate = 128000
Target Bit Rate = 128000
Frame Time = 6000
SQCIF MPI = 2
QCIF MPI = 2
CIF MPI = 2
4CIF MPI = 5
16CIF MPI = 5
Annex D = 1
Note that as this particular H.263 implementation cannot do custom frame
sizes, so all the frame sizes are adjusted appropriately. Some MPI's are set
to an illegally large value (required for merging to work) and the new frame
rate has made the MPI's that are left change upward.
This is encoded to:
m=video 5002 RTP/AVP 34
a=rtpmap:34 h263/90000
a=fmtp:34 CIF=2;QCIF=2;SQCIF=2;D=1
The remote replies:
m=video 5002 RTP/AVP 34
a=rtpmap:34 h263/90000
a=fmtp:34 CIF=3;4CIF=3
I deliberately tried to be a "rude" UA and return a frame size that was
never offered to make sure it all works.
OPAL then constructs the following from the SDP:
Max Frame Width = 640
Max Frame Height = 480
Min Frame Width = 144
Min Frame Height = 96
Frame Width = 320
Frame Height = 240
Max Bit Rate = 128000
Target Bit Rate = 128000
Frame Time = 6000
SQCIF MPI = 5
QCIF MPI = 5
CIF MPI = 3
4CIF MPI = 3
16CIF MPI = 5
Annex D = 0
Many of the above values are irrelevant at this stage and just inherited
from the master format.
OPAL then merges the sent options with the received options to get:
Max Frame Width = 352
Max Frame Height = 288
Min Frame Width = 144
Min Frame Height = 96
Frame Width = 176
Frame Height = 144
Max Bit Rate = 128000
Target Bit Rate = 128000
Frame Time = 6000
SQCIF MPI = 5
QCIF MPI = 5
CIF MPI = 3
4CIF MPI = 5
16CIF MPI = 5
Annex D = 0
Here the MaxMerge operator adjusts the MPI's, and the AndMerge operator
turns off Annex D.
This is then passed to the plug-in "to_normalised_options" getting:
Max Frame Width = 352
Max Frame Height = 288
Min Frame Width = 352
Min Frame Height = 288
Frame Width = 352
Frame Height = 288
Max Bit Rate = 128000
Target Bit Rate = 128000
Frame Time = 9009
SQCIF MPI = 5
QCIF MPI = 5
CIF MPI = 3
4CIF MPI = 5
16CIF MPI = 5
Annex D = 0
Where the Frame Size settles on its only possible value, and Frame Time gets
adjusted to 10fps due to CPI MPI being 3. These options are then sent to the
codec using set_codec_options and also merged with the YUV320P options so it
can be used by the grabber.
Robert Jongbloed
OPAL/OpenH323 Architect and Co-founder.
