I'm not sure why you don't get de-mangeld symbols in your backtrace and no line numbers, but from what I can see, your OpenMCU crashes, because it is accessing an invalid pointer that is used inside the RTP_JitterBuffer::ReadData() method in a PTRACE statement. This is definitely a bug in the code in either H323Plus or OpenMCU, probably the H323Plus.
Regards, Jan
Michael Miller wrote:
Here's what I get when I use the debug version of OpenMCU located in : h323plus_src/applications/openmcu/obj_linux_x86_d
You can see that I was able to connect a client and then the second client crashed OpenMCU. The first unit was a polycom vsx-8000 the second was a polycom vs-4000.
I'm guessing the error points to here: 00b61000-00ba8000 rw-p 0077d000 fd:00 450315 /root/h323plus_src/h323plus/lib/libh323_linux_x86__d.so.1.20-beta5
But I'm not sure what to do about it.
Any suggestions are greatly appreciated.
Michael
(gdb) set args -c -n -d --video --defaultroom ncsa1 (gdb) run Starting program: /root/h323plus_src/applications/openmcu/obj_linux_x86_d/openmcu -c -n -d --video --defaultroom ncsa1 [Thread debugging using libthread_db enabled] [New Thread 0xb7fea740 (LWP 12035)] error loading /usr/local/lib/pwlib/avcodec - /usr/local/lib/pwlib/avcodec: cannot open shared object file: No such file or directory error loading /usr/local/lib/pwlib/libavcodec.so - /usr/local/lib/pwlib/libavcodec.so: cannot open shared object file: No such file or directory All output for OpenMCU is to console. 2009/06/07 23:53:11.441 OpenMCU Message Starting service process "OpenMCU" v2.2.4 Detaching after fork from child process 12044. Daemon started with pid 12044 2009/06/07 23:53:11.451 OpenMCU Debug3 PWLib File handle high water mark set: 8 Thread unblock pipe 2009/06/07 23:53:11.453 OpenMCU Debug3 PTLib Thread high water mark set: 2 2009/06/07 23:53:11.458 OpenMCU Debug3 PWLib File handle high water mark set: 10 Thread unblock pipe 2009/06/07 23:53:11.459 OpenMCU Debug3 PTLib Thread high water mark set: 3 2009/06/07 23:53:11.464 OpenMCU Debug3 2009/06/07 23:53:11.468 OpenMCU Debug3 PWLib File handle high water mark set: 12 Thread unblock pipe 2009/06/07 23:53:11.471 OpenMCU Debug3 PTLib Thread high water mark set: 4 2009/06/07 23:53:11.472 H323 Cleaner:0xb7f67b90 Debug3 H323 Started cleaner thread 2009/06/07 23:53:11.474 OpenMCU Debug3 H323 Created endpoint. 2009/06/07 23:53:11.476 OpenMCU Debug3 PWLib File handle high water mark set: 14 Thread unblock pipe 2009/06/07 23:53:11.478 OpenMCU Debug3 PTLib Thread high water mark set: 5 2009/06/07 23:53:11.479 PXConfigWr...0xb7f26b90 Debug3 PTLib Config file cache write back thread started. 2009/06/07 23:53:11.480 OpenMCU Debug3 PTLib Created PXConfig 0x8221048 2009/06/07 23:53:11.481 OpenMCU Debug3 PTLib Reading config file: /root/.pwlib_config/openmcu.ini
Program exited normally. Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.4-5.fc10.i386 glibc-2.9-3.i686 keyutils-libs-1.2-3.fc9.i386 krb5-libs-1.6.3-18.fc10.i386 libgcc-4.3.2-7.i386 libselinux-2.0.78-1.fc10.i386 libstdc++-4.3.2-7.i386 openssl-0.9.8g-13.fc10.i686 zlib-1.2.3-18.fc9.i386 (gdb) 2009/06/07 23:53:11.484 OpenMCU Debug3 PWLib File handle high water mark set: 15 PTextFile 2009/06/07 23:53:11.489 OpenMCU Debug3 2009/06/07 23:53:11.701 OpenMCU Info Registered with Gatekeeper: 141.142.222.234 2009/06/07 23:53:11.718 OpenMCU Info Opened master socket for HTTP: 1420 2009/06/07 23:53:11.724 OpenMCU Info Service OpenMCU Started Adaptive Packet Delay = 0 - 0 Annex D = 0 - 0 CIF MPI = 1 - 1 CIF16 MPI = 0 - 0 CIF4 MPI = 0 - 0 Dynamic Video Quality = 0 - 0 Encoding Quality = 15 - 15 Frame Height = 288 - 288 Frame Time = 3003 - 3003 Frame Width = 352 - 352 h323_stillImageTransmission = 1 - 1 h323_temporalSpatialTradeOffCapability = 0 - 0 Max Bit Rate = 621700 - 621700 QCIF MPI = 0 - 0 SQCIF MPI = 0 - 0 Target Bit Rate = 64000 - 64000 Adaptive Packet Delay = 0 - 0 Annex D = 0 - 0 CIF MPI = 1 - 1 CIF16 MPI = 0 - 0 CIF4 MPI = 0 - 0 Dynamic Video Quality = 0 - 0 Encoding Quality = 15 - 15 Frame Height = 288 - 288 Frame Time = 3003 - 3003 Frame Width = 352 - 352 h323_stillImageTransmission = 1 - 1 h323_temporalSpatialTradeOffCapability = 0 - 0 Max Bit Rate = 621700 - 621700 QCIF MPI = 0 - 0 SQCIF MPI = 0 - 0 Target Bit Rate = 64000 - 64000 *** glibc detected *** /root/h323plus_src/applications/openmcu/obj_linux_x86_d/openmcu: malloc(): memory corruption (fast): 0xb7d83780 *** ======= Backtrace: ========= /lib/libc.so.6[0x2553a4] /lib/libc.so.6[0x258356] /lib/libc.so.6(__libc_malloc+0x95)[0x259765] /usr/lib/libstdc++.so.6(_Znwj+0x27)[0x3b40297] /usr/local/lib/libpt_d.so.2.6.2(_ZN13PStringStreamC1Ev+0x38)[0xe8044a] /usr/local/lib/libpt_d.so.2.6.2(_ZN6PTrace5BeginEjPKci+0xdd)[0xe656c5] /usr/local/lib/libh323_linux_x86__d.so.1.20-beta5(_ZN16RTP_JitterBuffer8ReadDataEjR13RTP_DataFrame+0x9f9)[0xb1e997] /usr/local/lib/libh323_linux_x86__d.so.1.20-beta5(_ZN11RTP_Session16ReadBufferedDataEjR13RTP_DataFrame+0x36)[0xa176ec] /usr/local/lib/libh323_linux_x86__d.so.1.20-beta5(_ZN15H323_RTPChannel7ReceiveEv+0x833)[0xa0774f] /usr/local/lib/libh323_linux_x86__d.so.1.20-beta5(_ZN24H323LogicalChannelThread4MainEv+0xa5)[0xa04e7f] /usr/local/lib/libpt_d.so.2.6.2(_ZN7PThread14PX_ThreadStartEPv+0x154)[0xe2c226] /lib/libpthread.so.0[0x39451f] /lib/libc.so.6(clone+0x5e)[0x2ca04e] ======= Memory map: ======== 00110000-00123000 r-xp 00000000 fd:00 51594 /lib/libresolv-2.9.so 00123000-00124000 r--p 00012000 fd:00 51594 /lib/libresolv-2.9.so 00124000-00125000 rw-p 00013000 fd:00 51594 /lib/libresolv-2.9.so 00125000-00127000 rw-p 00125000 00:00 0 00127000-00129000 r-xp 00000000 fd:00 51606 /lib/libcom_err.so.2.1 00129000-0012a000 rw-p 00001000 fd:00 51606 /lib/libcom_err.so.2.1 0012a000-00133000 r-xp 00000000 fd:00 26804 /usr/local/lib/pwlib/codecs/audio/lpc10_audio_pwplugin.so 00133000-00134000 rw-p 00008000 fd:00 26804 /usr/local/lib/pwlib/codecs/audio/lpc10_audio_pwplugin.so 00134000-00188000 r-xp 00000000 fd:00 26807 /usr/local/lib/pwlib/codecs/audio/gsmamrcodec_pwplugin.so 00188000-0018f000 rw-p 00054000 fd:00 26807 /usr/local/lib/pwlib/codecs/audio/gsmamrcodec_pwplugin.so 0018f000-00190000 r-xp 00000000 fd:00 26806 /usr/local/lib/pwlib/codecs/audio/ima_adpcm_audio_pwplugin.so 00190000-00191000 rw-p 00000000 fd:00 26806 /usr/local/lib/pwlib/codecs/audio/ima_adpcm_audio_pwplugin.so 00191000-00194000 r-xp 00000000 fd:00 26805 /usr/local/lib/pwlib/codecs/audio/g726_audio_pwplugin.so 00194000-00195000 rw-p 00003000 fd:00 26805 /usr/local/lib/pwlib/codecs/audio/g726_audio_pwplugin.so 00195000-001a2000 r-xp 00000000 fd:00 26803 /usr/local/lib/pwlib/codecs/audio/ilbc_audio_pwplugin.so 001a2000-001a5000 rw-p 0000c000 fd:00 26803 /usr/local/lib/pwlib/codecs/audio/ilbc_audio_pwplugin.so 001ab000-001b8000 r-xp 00000000 fd:00 51599 /lib/libgcc_s-4.3.2-20081105.so.1 001b8000-001b9000 rw-p 0000c000 fd:00 51599 /lib/libgcc_s-4.3.2-20081105.so.1 001bb000-001bd000 r-xp 00000000 fd:00 51605 /lib/libkeyutils-1.2.so 001bd000-001be000 rw-p 00001000 fd:00 51605 /lib/libkeyutils-1.2.so 001c1000-001e1000 r-xp 00000000 fd:00 51582 /lib/ld-2.9.so 001e2000-001e3000 r--p 00020000 fd:00 51582 /lib/ld-2.9.so 001e3000-001e4000 rw-p 00021000 fd:00 51582 /lib/ld-2.9.so 001e6000-00354000 r-xp 00000000 fd:00 51583 /lib/libc-2.9.so 00354000-00356000 r--p 0016e000 fd:00 51583 /lib/libc-2.9.so 00356000-00357000 rw-p 00170000 fd:00 51583 /lib/libc-2.9.so 00357000-0035a000 rw-p 00357000 00:00 0 0035c000-00383000 r-xp 00000000 fd:00 51596 /lib/libm-2.9.so 00383000-00384000 r--p 00026000 fd:00 51596 /lib/libm-2.9.so 00384000-00385000 rw-p 00027000 fd:00 51596 /lib/libm-2.9.so 00387000-0038a000 r-xp 00000000 fd:00 51589 /lib/libdl-2.9.so 0038a000-0038b000 r--p 00002000 fd:00 51589 /lib/libdl-2.9.so 0038b000-0038c000 rw-p 00003000 fd:00 51589 /lib/libdl-2.9.so 0038e000-003a4000 r-xp 00000000 fd:00 51584 /lib/libpthread-2.9.so 003a4000-003a5000 r--p 00015000 fd:00 51584 /lib/libpthread-2.9.so 003a5000-003a6000 rw-p 00016000 fd:00 51584 /lib/libpthread-2.9.so 003a6000-003a8000 rw-p 003a6000 00:00 0 003aa000-003bd000 r-xp 00000000 fd:00 49286 /lib/libz.so.1.2.3 003bd000-003be000 rw-p 00012000 fd:00 49286 /lib/libz.so.1.2.3 003c0000-003db000 r-xp 00000000 fd:00 51591 /lib/libselinux.so.1 003db000-003dc000 r--p 0001a000 fd:00 51591 /lib/libselinux.so.1 003dc000-003dd000 rw-p 0001b000 fd:00 51591 /lib/libselinux.so.1 003e3000-00b61000 r-xp 00000000 fd:00 450315 /root/h323plus_src/h323plus/lib/libh323_linux_x86__d.so.1.20-beta5 00b61000-00ba8000 rw-p 0077d000 fd:00 450315 /root/h323plus_src/h323plus/lib/libh323_linux_x86__d.so.1.20-beta5 00ba8000-00baa000 rw-p 00ba8000 00:00 0 00baa000-00bcb000 r-xp 00000000 fd:00 26802 /usr/local/lib/pwlib/codecs/audio/speex_audio_pwplugin.so 00bcb000-00bcd000 rw-p 00020000 fd:00 26802 /usr/local/lib/pwlib/codecs/audio/speex_audio_pwplugin.so 00bcd000-00bd6000 r-xp 00000000 fd:00 26801 /usr/local/lib/pwlib/codecs/audio/gsm0610_audio_pwplugin.so 00bd6000-00bd7000 rw-p 00008000 fd:00 26801 /usr/local/lib/pwlib/codecs/audio/gsm0610_audio_pwplugin.so 00bd7000-00be2000 r-xp 00000000 fd:00 49242 /lib/libnss_files-2.9.so 00be2000-00be3000 r--p 0000a000 fd:00 49242 /lib/libnss_files-2.9.so 00be3000-00be4000 rw-p 0000b000 fd:00 49242 /lib/libnss_files-2.9.so 00bfb000-00e98000 r-xp 00000000 fd:00 435104 /root/ptlib-2.6.2/lib_linux_x86/libpt_d.so.2.6.2 00e98000-00ec0000 rw-p 0029d000 fd:00 435104 /root/ptlib-2.6.2/lib_linux_x86/libpt_d.so.2.6.2 00ec0000-00ec3000 rw-p 00ec0000 00:00 0 00ec3000-00efe000 r-xp 00000000 fd:00 199303 /usr/local/lib/pwlib/codecs/video/h261-vic_video_pwplugin.so 00efe000-00f0c000 rw-p 0003b000 fd:00 199303 /usr/local/lib/pwlib/codecs/video/h261-vic_video_pwplugin.so 00f2b000-00f31000 r-xp 00000000 fd:00 199304 /usr/local/lib/pwlib/codecs/video/h263-ffmpeg_video_pwplugin.so 00f31000-00f32000 rw-p 00006000 fd:00 199304 /usr/local/lib/pwlib/codecs/video/h263-ffmpeg_video_pwplugin.so 00f71000-00f72000 r-xp 00f71000 00:00 0 [vdso] 037d9000-037fd000 r-xp 00000000 fd:00 41953 /usr/lib/libk5crypto.so.3.1 037fd000-037fe000 rw-p 00024000 fd:00 41953 /usr/lib/libk5crypto.so.3.1 03800000-03937000 r-xp 00000000
"If you're clear in your vision and trust the people in your team with clear objectives, they will invariably do their best to achieve everything desired, and usually deliver everything you could have hoped for and even more." -Paul Debevec
----- Original Message ----- From: "Jan Willamowius" jan@willamowius.de To: h323plus@lists.packetizer.com Sent: Saturday, June 6, 2009 2:34:37 PM GMT -06:00 US/Canada Central Subject: Re: [h323plus] OpenMCU shutting down unexpectedly
Hi Michael,
I think "make install" always installs the optimized executable. I would enable core dumps ("ulimit -c unlimited") and then execute the debug version of openmcu right out of the compile directory (./obj_linux_x86_64_d/openmcu ....").
When you then crash openmcu, you can start gdb with "gdb ./obj_linux_x86_64_d/openmcu core" and execute a "bt" command to see where it crashed.
Regards, Jan
Michael Miller wrote:
Jan,
I'm not sure I did this right. I went to the ptlib-2.6.2 directory and executed: make debug make install
That all went fine. Then I moved to the openmcu directory and executed: make debug make install
That all appears to work fine. But when I go to trace it I get:
~]# gdb /bin/openmcu GNU gdb Fedora (6.8-29.fc10) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"... (no debugging symbols found) (gdb) set args -c -n -d --video --defaultroom ncsa1 (gdb) run Starting program: /bin/openmcu -c -n -d --video --defaultroom ncsa1 (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [New Thread 0xb7fea740 (LWP 6526)] error loading /usr/local/lib/pwlib/avcodec - /usr/local/lib/pwlib/avcodec: cannot open shared object file: No such file or directory error loading /usr/local/lib/pwlib/libavcodec.so - /usr/local/lib/pwlib/libavcodec.so: cannot open shared object file: No such file or directory All output for OpenMCU is to console. 2009/06/02 04:39:14.431 OpenMCU Message Starting service process "OpenMCU" v2.2.4 Detaching after fork from child process 6535. Daemon started with pid 6535 2009/06/02 04:39:14.456 OpenMCU Debug3 2009/06/02 04:39:14.465 OpenMCU Debug3 PWLib File handle high water mark set: 8 Thread unblock pipe 2009/06/02 04:39:14.466 OpenMCU Debug3 PTLib Thread high water mark set: 2
Program exited normally. Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.4-5.fc10.i386 glibc-2.9-3.i686 keyutils-libs-1.2-3.fc9.i386 krb5-libs-1.6.3-18.fc10.i386 libgcc-4.3.2-7.i386 libselinux-2.0.78-1.fc10.i386 libstdc++-4.3.2-7.i386 openssl-0.9.8g-13.fc10.i686 zlib-1.2.3-18.fc9.i386 (gdb) 2009/06/02 04:39:14.470 OpenMCU Debug3 PWLib File handle high water mark set: 10 Thread unblock pipe 2009/06/02 04:39:14.470 OpenMCU Debug3 PTLib Thread high water mark set: 3 2009/06/02 04:39:14.475 OpenMCU Debug3 PWLib File handle high water mark set: 12 Thread unblock pipe 2009/06/02 04:39:14.476 OpenMCU Debug3 PTLib Thread high water mark set: 4 2009/06/02 04:39:14.477 OpenMCU Debug3 H323 Created endpoint. 2009/06/02 04:39:14.479 OpenMCU Debug3 PWLib File handle high water mark set: 14 Thread unblock pipe 2009/06/02 04:39:14.480 H323 Cleaner:0xb7f67b90 Debug3 H323 Started cleaner thread 2009/06/02 04:39:14.480 OpenMCU Debug3 PTLib Thread high water mark set: 5 2009/06/02 04:39:14.481 PXConfigWr...0xb7dffb90 Debug3 PTLib Config file cache write back thread started. 2009/06/02 04:39:14.482 OpenMCU Debug3 PTLib Created PXConfig 0x81fdd68 2009/06/02 04:39:14.482 OpenMCU Debug3 PTLib Reading config file: /root/.pwlib_config/openmcu.ini 2009/06/02 04:39:14.483 OpenMCU Debug3 PWLib File handle high water mark set: 15 PTextFile 2009/06/02 04:39:14.494 OpenMCU Debug3 2009/06/02 04:39:14.793 OpenMCU Info Registered with Gatekeeper: 141.142.222.234 2009/06/02 04:39:14.801 OpenMCU Info Opened master socket for HTTP: 1420 2009/06/02 04:39:14.802 OpenMCU Info Service OpenMCU Started
I'm not sure what the "(no debugging symbols found)" is referring to...
Any suggestions are greatly appreciated.
Michael
"If you're clear in your vision and trust the people in your team with clear objectives, they will invariably do their best to achieve everything desired, and usually deliver everything you could have hoped for and even more." -Paul Debevec
----- Original Message ----- From: "Jan Willamowius" jan@willamowius.de To: h323plus@lists.packetizer.com Sent: Tuesday, June 2, 2009 12:55:28 PM GMT -06:00 US/Canada Central Subject: Re: [h323plus] OpenMCU shutting down unexpectedly
Hi,
the compile will work if you compile PTLib directly with 'make debug' and don't rely on the OpenMCU makefile to start the compilation of PTLib.
Regards, Jan
Michael Miller wrote:
Hello,
I have OpenMCU compiled/installed from the latest cvs. When I get a third client connected, OpenMCU shuts down unexpectedly. It appears to get a shutdown signal from somewhere. I have attached an strace. Let me know if more info is needed.
Also, in pursuing more info about this, I tried compiling OpenMCU with the debug option and got:
]# make debug make DEBUG=1 default_target make[1]: Entering directory `/root/openmcu-v2_2_4-src/applications/openmcu' g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c main.cxx -o obj_linux_x86_d/main.o g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c video.cxx -o obj_linux_x86_d/video.o video.cxx: In member function ‘PBoolean MCUSimpleVideoMixer::ReadSrcFrame(VideoFrameStoreList&, void*, int, int, PINDEX&)’: video.cxx:953: warning: comparison between signed and unsigned integer expressions video.cxx:953: warning: comparison between signed and unsigned integer expressions g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c conference.cxx -o obj_linux_x86_d/conference.o g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c filemembers.cxx -o obj_linux_x86_d/filemembers.o g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c custom.cxx -o obj_linux_x86_d/custom.o g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c h323.cxx -o obj_linux_x86_d/h323.o h323.cxx: In member function ‘virtual PBoolean OpenMCUH323Connection::OnIncomingAudio(const void*, PINDEX)’: h323.cxx:926: warning: comparison is always false due to limited range of data type g++ -I/root/pwlib/include -DPTRACING=1 -D_REENTRANT -I/usr/kerberos/include -fno-exceptions -Wall -g3 -ggdb -O0 -D_DEBUG -DPTRACING -I/root/h323plus/include -felide-constructors -Wreorder -c mcu.cxx -o obj_linux_x86_d/mcu.o make -C /root/pwlib/src/ptlib/unix debug make[2]: Entering directory `/root/ptlib-2.6.2/src/ptlib/unix' make[2]: *** No rule to make target `debug'. Stop. make[2]: Leaving directory `/root/ptlib-2.6.2/src/ptlib/unix' make[1]: *** [/root/pwlib/lib_linux_x86/libpt_d.so] Error 2 make[1]: Leaving directory `/root/openmcu-v2_2_4-src/applications/openmcu' make: *** [debug] Error 2
I have set: PTLIBDIR=/root/pwlib PTLIBPLUGINDIR=/usr/local/lib/pwlib/
Any other suggestions for making OpenMCU debuggable?
thanks Michael "If you're clear in your vision and trust the people in your team with clear objectives, they will invariably do their best to achieve everything desired, and usually deliver everything you could have hoped for and even more." -Paul Debevec
-- Jan Willamowius, jan@willamowius.de, http://www.willamowius.de/
-- Jan Willamowius, jan@willamowius.de, http://www.willamowius.de/