Hello,

We have found probably a bug in ptlib during testing the t38modem application (ptlib+h323plus+t38modem). We got this backtrace: root@maxi:/tmp# gdb /usr/sbin/t38modem core.6382.104 GNU gdb 6.4-debian Copyright 2005 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".

Failed to read a valid object file image from memory. Core was generated by `/usr/sbin/t38modem -p ttyc0,ttyc1,ttyc2,ttyc3,ttyc4,ttyc5,ttyc6,ttyc7,ttyc8,tty'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libresolv.so.2...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libldap_r.so.2...done. Loaded symbols for /usr/lib/libldap_r.so.2 Reading symbols from /usr/lib/liblber.so.2...done. Loaded symbols for /usr/lib/liblber.so.2 Reading symbols from /lib/libpthread.so.0...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /usr/lib/libsasl2.so.2...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libssl.so.0.9.8...done. Loaded symbols for /usr/lib/libssl.so.0.9.8 Reading symbols from /usr/lib/libcrypto.so.0.9.8...done. Loaded symbols for /usr/lib/libcrypto.so.0.9.8 Reading symbols from /usr/lib/libexpat.so.1...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /usr/lib/libSDL-1.2.so.0...done. Loaded symbols for /usr/lib/libSDL-1.2.so.0 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /usr/lib/libstdc++.so.6...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libgcc_s.so.1...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /usr/lib/libgnutls.so.12...done. Loaded symbols for /usr/lib/libgnutls.so.12 Reading symbols from /lib/ld-linux-x86-64.so.2...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/libasound.so.2...done. Loaded symbols for /usr/lib/libasound.so.2 Reading symbols from /usr/lib/libX11.so.6...done. Loaded symbols for /usr/lib/libX11.so.6 Reading symbols from /usr/lib/libXext.so.6...done. Loaded symbols for /usr/lib/libXext.so.6 Reading symbols from /usr/lib/libtasn1.so.2...done. Loaded symbols for /usr/lib/libtasn1.so.2 Reading symbols from /usr/lib/libgcrypt.so.11...done. Loaded symbols for /usr/lib/libgcrypt.so.11 Reading symbols from /usr/lib/libgpg-error.so.0...done. Loaded symbols for /usr/lib/libgpg-error.so.0 Reading symbols from /usr/lib/libXau.so.6...done. Loaded symbols for /usr/lib/libXau.so.6 Reading symbols from /lib/libnsl.so.1...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libnss_compat.so.2...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnss_nis.so.2...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 #0 0x00002b3e71de4312 in __gnu_cxx::__exchange_and_add () from /usr/lib/libstdc++.so.6 (gdb) bt full #0 0x00002b3e71de4312 in __gnu_cxx::__exchange_and_add () from /usr/lib/libstdc++.so.6 No symbol table info available. #1 0x00000000005c4ee1 in PAtomicInteger::operator++ (this=0x4) at critsec.h:243 No locals. #2 0x000000000077cf8a in PContainer (this=0x40cf2080, cont=@0xafc3c0) at ../common/contain.cxx:739 No locals. #3 0x000000000046fdbd in PAbstractArray (this=0x40cf2080, c=@0xafc3c0) at array.h:183 No locals. #4 0x000000000047a19b in PBaseArray (this=0x40cf2080) at array.h:390 No locals. #5 0x0000000000789315 in PCharArray (this=0x40cf2080) at array.h:733 No locals. #6 0x000000000077d571 in PString (this=0x40cf2080, str=@0xafc3c0) at contain.inl:241 No locals. #7 0x000000000075c559 in PThread::PX_ThreadEnd (arg=0xafc3b0) at tlibthrd.cxx:1402 threadName = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x870d90}, reference = 0xf0}, elementSize = 0, theArray = 0x100000000 <Address 0x100000000 out of bounds>, allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>} process = (class PProcess &) @0xabc420: {<PThread> = {<PObject> = {_vptr.PObject = 0x7bfad0}, autoDelete = 0, threadName = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x8710b0}, reference = 0xabaf90}, elementSize = 1, theArray = 0xabc970 "", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, traceStream = {<PString> = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x871458}, reference = 0xac2530}, elementSize = 1, theArray = 0xabc990 "", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, <> = {<No data fields>}, <No data fields>}, traceLevel = 0, traceBlockIndentLevel = 0, PX_origStackSize = 0, PX_priority = PThread::NormalPriority, PX_threadId = 47547219615632, PX_suspendMutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0}, __size = '\0' <repeats 39 times>, __align = 0}, PX_suspendCount = 0, PX_firstTimeStart = 0, ending = 0, unblockPipe = {3, 4}}, static p_argc = 21, static p_argv = 0x7fff39b8a398, static p_envp = 0x7fff39b8a448, terminationValue = 0, manufacturer = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = { _vptr.PObject = 0x8710b0}, reference = 0xabc9b0}, elementSize = 1, theArray = 0xabcb40 "OpenH323 Project", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, productName = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x8710b0}, reference = 0xabcb60}, elementSize = 1, theArray = 0xabcb80 "T38Modem", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, majorVersion = 1, minorVersion = 0, status = PProcess::ReleaseCode, buildNumber = 0, executableFile = {<PString> = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x8691d0}, reference = 0xabcba0}, elementSize = 1, theArray = 0xabcbc0 "", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, <No data fields>}, configurationPaths = {<PList<PString>> = {<PAbstractList> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x871670}, reference = 0xabcbe0}, <No data fields>}, info = 0xabcc00}, <No data fields>}, <No data fields>}, arguments = {<PObject> = {_vptr.PObject = 0x86f190}, argumentArray = {<PArray<PString>> = {<PArrayObjects> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x871590}, reference = 0xabcee0}, <No data fields>}, theArray = 0xabcf00}, <No data fields>}, <No data fields>}, optionLetters = {<PCharArray> = {<PBaseArray<char>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x8710b0}, reference = 0xabcc30}, elementSize = 1, theArray = 0xabcc80 "", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, <No data fields>}, optionNames = {<PArray<PString>> = {<PArrayObjects> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x871590}, reference = 0xabcce0}, <No data fields>}, theArray = 0xabcd00}, <No data fields>}, <No data fields>}, optionCount = {<PBaseArray<int>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x7bfbf0}, reference = 0xabcd50}, elementSize = 4, theArray = 0x0, allocatedDynamically = 1}, <No data fields>}, <No data fields>}, optionString = {<PArray<PString>> = {<PArrayObjects> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x871590}, reference = 0xabcd70}, <No data fields>}, theArray = 0xabcd90}, <No data fields>}, <No data fields>}, parameterIndex = {<PBaseArray<int>> = {<PAbstractArray> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x7bfbf0}, reference = 0xabcde0}, elementSize = 4, theArray = 0xabd960 "", allocatedDynamically = 1}, <No data fields>}, <No data fields>}, shift = 0}, timers = {<PList<PTimer>> = {<PAbstractList> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x86a0b0}, reference = 0xabce00}, <No data fields>}, info = 0xabce20}, <No data fields>}, listMutex = {<PSync> = {<PObject> = {_vptr.PObject = 0x869d90}, lockerId = 1092376928}, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 1, __spins = 0}, __size = '\0' <repeats 16 times>, "\001", '\0' <repeats 22 times>, __align = 0}}, processingMutex = {<PSync> = {<PObject> = {_vptr.PObject = 0x869d90}, lockerId = 1092376928}, mutex = {__data = { __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 1, __spins = 0}, __size = '\0' <repeats 16 times>, "\001", '\0' <repeats 22 times>, __align = 0}}, inTimeoutMutex = {<PSync> = {<PObject> = {_vptr.PObject = 0x869d90}, lockerId = 1092376928}, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 1, __spins = 0}, __size = '\0' <repeats 16 times>, "\001", '\0' <repeats 22 times>, __align = 0}}, lastSample = {<PObject> = {_vptr.PObject = 0x868990}, milliseconds = 1203507183364}, currentTimer = 0x0}, programStartTime = {<PObject> = {_vptr.PObject = 0x868a10}, theTime = 1203506350, microseconds = 785187}, maxHandles = 2048, pxSignals = 0, configFiles = 0xabdab0, activeThreads = {<PAbstractDictionary> = {<PHashTable> = {<PCollection> = {<PContainer> = {<PObject> = {_vptr.PObject = 0x869f10}, reference = 0xabce50}, <No data fields>}, hashTable = 0xabce70}, <No data fields>}, <No data fields>}, threadMutex = {<PSync> = {<PObject> = {_vptr.PObject = 0x869d90}, lockerId = 1087318368}, mutex = {__data = {__lock = 2, __count = 1, __owner = 21470, __nusers = 1, __kind = 1, __spins = 0}, __size = "\002\000\000\000\001\000\000\000�S\000\000\001\000\000\000\001", '\0' <repeats 22 times>, __align = 4294967298}}, breakBlock = {<PSemaphore> = {<PSync> = {<PObject> = {_vptr.PObject = 0x869e10}, lockerId = 18446744073709551615}, initialVar = 0, maxCountVar = 0, pxClass = PSemaphore::PXSyncPoint, mutex = {__data = { __lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0}, __size = '\0' <repeats 12 times>, "\001", '\0' <repeats 26 times>, __align = 0}, condVar = {__data = {__lock = 0, __futex = 1659, __total_seq = 830, __wakeup_seq = 829, __woken_seq = 829, __mutex = 0xabc8d8, __nwaiters = 2, __broadcast_seq = 0}, __size = "\000\000\000\000{\006\000\000>\003\000\000\000\000\000\000=\003\000\000\000\000\000\000=\003\000\000\000\000\000\000�ȫ\000\000\000\000\000\002\000\000\000\000\000\000",

__align = 7125350744064}, semId = {__size = '\0' <repeats 31 times>, __align = 0}}, signalCount = 0}, housekeepingThread = 0xaedfc0} thread = (PThread *) 0xafc3b0 id = 1087318368 #8 0x000000000075c35e in PThread::PX_ThreadStart (arg=0xafc3b0) at tlibthrd.cxx:1364 __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {7805982, 1087316448, 47547187568752, 1091568592, 0, 1091568592, 1087316224, 7717523}, __mask_was_saved = 0}}, __pad = {0x40cf2200, 0x0, 0x1, 0x2b3e714a46c0}} ---Type <return> to continue, or q <return> to quit--- __cancel_routine = (void (*)(void *)) 0x75c36e PThread::PX_ThreadEnd(void*) __cancel_arg = (void *) 0xafc3b0 not_first_call = 0 thread = (PThread *) 0xafc3b0 #9 0x00002b3e7139a0fa in start_thread () from /lib/libpthread.so.0 No symbol table info available. #10 0x00002b3e72287ce2 in clone () from /lib/libc.so.6 No symbol table info available. #11 0x0000000000000000 in ?? () No symbol table info available. (gdb)

After long debugging we have found that the issue is in ptlib and the following patch fixes it: --- ptlib_v2_0_1/src/ptlib/unix/tlibthrd.cxx.orig 2008-03-19 22:13:56.000000000 +0100 +++ ptlib_v2_0_1/src/ptlib/unix/tlibthrd.cxx 2008-03-20 09:25:41.000000000 +0100 @@ -1387,10 +1387,11 @@

// delete the thread if required, note this is done this way to avoid // a race condition, the thread ID cannot be zeroed before the if! + PString threadName = thread->threadName; if (thread->autoDelete) { thread->PX_threadId = 0; // Prevent terminating terminated thread process.threadMutex.Signal(); - PTRACE(5, "PWLib\tEnded thread " << thread << ' ' << thread->threadName); + PTRACE(5, "PWLib\tEnded thread " << thread << ' ' << threadName);

/* It is now safe to delete this thread. Note that this thread is deleted after the process.threadMutex.Signal(), which means @@ -1399,7 +1400,6 @@ } else { thread->PX_threadId = 0; - PString threadName = thread->threadName; process.threadMutex.Signal(); PTRACE(5, "PWLib\tEnded thread " << thread << ' ' << threadName); }

It seems to be a race condition similar to what have been resolved earlier (see the comments and cvs history). After testing the patched version the issue seems to be over.

I would like thank Simon for helping with the debugging and taking care of t38modem for h323plus.

Kind regards, Tamas

ps: system info: ptlib-2.0.1, h323plus-1.20.2, t38modem from h323plus contrib. Ubuntu Dapper Drake LTS (6.06), x86_64, Linux maxi 2.6.23.9 #1 SMP PREEMPT Thu Dec 6 11:32:16 CET 2007 x86_64 GNU/Linux