[Robustness] DDP Checkpointing

Randall Stewart rstewar1 at EMAIL.MOT.COM
Wed Apr 12 17:05:03 EDT 2000


Terry:

I am also copying the sigtran list, we should be releasing
a new version of the draft with NO technical changes shortly...
this will have just a new "Motiviation" section to help
folks understand why we built DDP and where it fits... Hopefully
this will bring about discussion of this on the sigtran list.

Sigtraners: If you are interested in DDP a bit, read on, if not
you can skip this email... R

I have contacted the Application Area AD's (per the suggestions
at the last IETF) and they have as yet not responded...

Let me detail some of the state idea's we have done
in our testbed. Sorry all that I missed the conf
call today... alergy season had me under the weather :-0

Now has you state DDP itself does NOT provide any state
sharing mechanism per se. There is a mechanism called
SEND_TO_ALL, which sends to every one of a given named
group (except myself if I am of that same name, which you
would be in the case of a state sharing situation). This
will probably need to be extended with one more additional
type, SEND_TO_BACKUP's. This would be in conjunction with
the new load balancing algrithm that we need to add to
have failures go to guy's designated has backup.

In itself these are very primitive mechanisms for state sharing
and don't scale to well. However what we did is create a
set of "distributed shared memory".

Our mechanism is built on two foundations:
 o The application wishing to share state includes a library, in
   our case it is a object since we are coding in C++ but it could
   just as well be a c library.

 o A deamon that runs on every machine that wishes to share state.
   Note there is only ONE deamon per physical machine.

Now all the deamons register under a specific name, for my
example I will call the shared memory "terry". So all of our
"terry" deamon register and each allocates a bunch of shared
memory on there local O/S.

When a application process starts up on the local machine it
initializes its object (or library) by giving it the name of
the shared memory deamon i.e. "terry". This allows the library/object
to find the shared memory and communicate with the deamon (via
the object, the application is never directly aware of communications).

To the local application the object/library appears to be much
like a hash table. It has the following primitives:

allocate()
push()
lookup()
remove()

allocate has you might imagine allocates a piece of shared memory
from the shared memory pool via a specific key. The key can be
anything the app wants..just like a hash table... lets say
we use a call-reference-number. Now the application now, as
a result of the allocate, gets a pointer to a piece of
local memory. It puts some state in this block and when
it decides to checkpoint it does a push().. The push is
rather flexible allowing it to push the whole thing OR
push just a specific set of bytes of that piece of memory.

When the push happens notifications go out between the library/object
and deamon. The deamon will then take and make sure that ALL other
deamons gets a copy of this memory.

Now in theory (in our model at least) the call stays attached to
the guy that allocated the memory, unless he fails. Has such we
stress to our developers that they probably should ONLY use the
minimum memory to reconstruct state in this shared memory block
and NOT use it has there call control block, they could/can
violate this principle, but the idea here is to have a recovery
mechanism not a place to put your call state :-)

If my local sever fails, the DDP is optioned by its clients to
"fail-over" to some other call server. When this occurs our
friend the peer call server sees a call-reference-number that
it knows nothing about. So what does it do? it does a lookup()
call on that key. If it finds nothing.. we just lost the call..
OR hopefully we find the state block, reconstruct the call state
and thus the current call object and continue the call...

Now notice there is NO synchronization between the shared memory
deamons in my example. This case was not deemed worth the overhead
since we use the "lazy binding" principle that you stay with the
first call server that serves you on your call unless he fails.. Other
models of servers are possible that would do 2-phase commit type
approaches to the shared memory if it was required...

So this is how we share state in DDP. DDP only provides the framework
for building this and the tools for the server/client to transparently
fail over with minimal work...


Now let me see if I can answer anything I missed in your email....

Terry L Anderson wrote:
>
> Randy Stewart -
>
> You described a "checkpointing" mechanism for DDP during last week's
> robustness call but it is not in the current DDP draft.  However, you
> have evidently thought about this some and stated (in private mail) that
> you had built such mechanism on top of DDP using the SEND_TO_ALL
> mechanism.  How would you envision something like this being done to
> synch state for a pool of H.323 entities?  Would this be a application
> level library that could be called from H.323 software at appropriate
> checkpoint times?  Could you describe this checkpointing idea in some
> more detail?

I think the above covered this.. let me know if I am unclear.. still
have a sinus headache so I may be a bit vague...

>
> It would seem preferable to either break up state information into a
> number of subsets (to avoid resending large amounts each time) or to
> only send differences.  Since DDP gives reliable delivery to the other
> members of the pool, perhaps only changes need be sent.  Did your trials
> solve this problem.  Perhaps each call would be a separate subset.

Well in effect we only do the push() above at the point we consider
it a stable call. We did discuss doing calls in setup stages but
decided that we did not need to do this.. it is possible since
our push() mechanism lets us push selected pieces but we currently
only do it for the whole thing when the calls are stable...

>
> One issue is how to solve reliable end-to-end delivery.  SCTP would
> guarantee delivery to other end of connection but  the H.225 and H.245
> protocols may pass through intermediate nodes (e.g., routing GKs).  If a
> node fails after acknowledging the message but before sending it on,
> end-to-end delivery fails and no live element knows.  Either we need far
> end acknowledgement or backup elements must accept delivery
> reponsibility once acknowledgement has been sent by its later failed
> peer.  DDP would guarantee delivery but the recipient would have to
> checkpoint this message before acknowledgement to guarantee that a peer
> taking over whould know that there was an outstanding message that it
> must deliver and receive acknowledgement for.  This mechanism would
> require checkpoint after receipt of but before sending acknowledgement
> of a message and again after receiving acknowledgement from its next
> neighbor.  Does this seem reasonable?  Unless the amount of data sent in
> such checkpointing is kept very small this would seem to add too much
> network activity to be acceptible.

I think this is a reasonable method, depending on how much data is
in the checkpoint. The key is to keep the checkpointed data small
and also the range of the shared memory distribution..


>
> --
> ------------------------------------------------------------
> Terry L Anderson              mailto:tla at lucent.com
> Tel:908.582.7013   Fax:908.582.6729
> Pager:800.759.8352 pin 1704572   1704572 at skytel.com
> Lucent Technologies/ Voice Over IP Access Networks/ Applications Grp
> Rm 2B-121, 600 Mountain Av, Murray Hill, NJ 07974
> http://its.lucent.com/~tla (Lucent internal) http://www.gti.net/tla


If I have missed anything let me know...

--
Randall R. Stewart
Member Technical Staff
Network Architecture and Technology (NAT)
847-632-7438 fax:847-632-6733



More information about the sg16-avd mailing list