[SERVER-10992] client / secondExample failing on Linux 32-bit (MCI only, not Buildbot) Created: 30/Sep/13  Updated: 11/Jul/16  Resolved: 13/Oct/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 2.5.3

Type: Bug Priority: Blocker - P1
Reporter: Matt Kangas Assignee: Andrew Morrow (Inactive)
Resolution: Done Votes: 1
Labels: buildbot
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-11130 authTest-sharedclient failing on RHEL... Closed
Operating System: ALL
Participants:

 Description   

http://mci.10gen.com/ui/task/mongodb_mongo_master_linux_32_d72362a3ecb69f8f504b2b99a0adf64ae5cf7a12_13_09_29_13_12_06_client_linux_32
http://buildlogs.mongodb.org/build/52483dd4d2a60f6f04000240/test/524842c2d2a60f187b000466/

The test output is remarkably abrupt:

using cursor
	{ "_id" : { "$oid" : "524842c21ef1a8b6e75f1902" }, "name" : "eliot", "num" : 17 }
	{ "_id" : { "$oid" : "524842c21ef1a8b6e75f1903" }, "name" : "sara", "num" : 24 }
2013-09-29T15:09:54.730+0000 [conn7] end connection 127.0.0.1:36748 (0 connections now open)
2013-09-29 11:09:59 EDT	
2013-09-29T15:09:59.447+0000 [initandlisten] connection accepted from 127.0.0.1:36790 #8 (1 connection now open)

The scons output says it completes in 204.4649 ms

Curiously, Buildbot is green on Linux-32 right now. Here's Buildbot's output for the same test.

http://buildbot.mongodb.org/builders/Linux%2032-bit/builds/6001/steps/smokeClient/logs/stdio
http://buildlogs.mongodb.org/Linux%2032-bit/builds/6001/test/client/secondExample

The test started failing on MCI on Fri Sept 13. First visible failure was on commit 9ab28c9b by Eric. But immediately before that was:

FAIL:     9ab28c9 2013-09-10 17:11 Eric Milkie        o SERVER-9771 build Windows dynamic library C++ driver
UNTESTED: 8214ffd 2013-09-10 17:11 Eric Milkie        o SERVER-9771 allow libdeps to process emitter intermed~
UNTESTED: 887aa5b 2013-09-12 15:15 Jason Rassi        o SERVER-10650 Add new fail point maxTimeAlwaysTimeOut
UNTESTED: 63f777e 2013-09-12 15:17 Jason Rassi        o SERVER-2212 Separate processing of "$maxTimeMS" from ~
UNTESTED: bf0055f 2013-09-12 15:16 Jason Rassi        o SERVER-2212 Minor block formatting in parsed_query.cpp
UNTESTED: 767a142 2013-09-12 14:46 Andrew Morrow      o SERVER-10159 Don't declare storage for static integra~
GOOD:     ed68745 2013-09-12 14:40 Spencer T Brody    o SERVER-9980 Small cleanup to write concern in user ma~



 Comments   
Comment by auto [ 13/Oct/13 ]

Author:

{u'username': u'acmorrow', u'name': u'Andrew Morrow', u'email': u'acm@10gen.com'}

Message: SERVER-10992 SERVER-11130 Avoid inclusion of boost symbols for non system boost client driver build
Branch: master
https://github.com/mongodb/mongo/commit/563657a6d65184d4efb3326a999bab2abcb083c2

Comment by Andrew Morrow (Inactive) [ 01/Oct/13 ]

This type of memory error does not always lead to a crash. Whether or not it will crash is closely tied to the exact libc and linker version in play. Unless the MCI machines are running images that are file-for-file identical to the buildbot machines there is no reason to expect it to appear consistently.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Thanks for the info., any ideas why this isn't failing on Buildbot?

Comment by Andrew Morrow (Inactive) [ 01/Oct/13 ]

Eric and I discussed, and we believe that we understand the root problem. It is related to the fact that in the real world, use of the shared library libmongoclient is always supposed to work with a shared library boost. However, our --sharedclient build uses the vendored boost. This, coupled with some LIBDEPS behavior, you end up with some symbols in both libmongoclient and secondExample, and their is a collision at static destruction time, causing some static std::string to be destroyed twice.

We need to think about how we want to tackle this; there is not an immediately obvious solution.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

With --track-origins=yes:

[ec2-user@ip-10-165-38-116 artifacts]$ valgrind --tool=memcheck --leak-check=yes --track-origins=yes ./sharedclient/secondExample --port 27999
==14330== Memcheck, a memory error detector
==14330== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==14330== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==14330== Command: ./sharedclient/secondExample --port 27999
==14330== 
using cursor
	{ "_id" : { "$oid" : "524b2204d57683f19fd004e6" }, "name" : "eliot", "num" : 17 }
	{ "_id" : { "$oid" : "524b2204d57683f19fd004e7" }, "name" : "sara", "num" : 24 }
==14330== Invalid read of size 4
==14330==    at 0x42B1AB3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (atomicity.h:48)
==14330==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14330==    by 0x43811F6: __cxa_finalize (cxa_finalize.c:56)
==14330==    by 0x4095913: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14330==    by 0x41A2488: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14330==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14330==    by 0x4380EEC: exit (exit.c:100)
==14330==    by 0x4368A0A: (below main) (libc-start.c:258)
==14330==  Address 0x45f6740 is 8 bytes inside a block of size 16 free'd
==14330==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14330==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14330==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14330==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14330==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14330==    by 0x4380EEC: exit (exit.c:100)
==14330==    by 0x4368A0A: (below main) (libc-start.c:258)
==14330== 
==14330== Invalid free() / delete / delete[] / realloc()
==14330==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14330==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14330==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14330==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14330==    by 0x43811F6: __cxa_finalize (cxa_finalize.c:56)
==14330==    by 0x4095913: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14330==    by 0x41A2488: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14330==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14330==    by 0x4380EEC: exit (exit.c:100)
==14330==    by 0x4368A0A: (below main) (libc-start.c:258)
==14330==  Address 0x45f6738 is 0 bytes inside a block of size 16 free'd
==14330==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14330==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14330==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14330==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14330==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14330==    by 0x4380EEC: exit (exit.c:100)
==14330==    by 0x4368A0A: (below main) (libc-start.c:258)
==14330== 
==14330== 
==14330== HEAP SUMMARY:
==14330==     in use at exit: 1,049,869 bytes in 43 blocks
==14330==   total heap usage: 461 allocs, 419 frees, 1,138,975 bytes allocated
==14330== 
==14330== 16 bytes in 1 blocks are definitely lost in loss record 13 of 42
==14330==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14330==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14330==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14330==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14330==    by 0x409573A: _GLOBAL__sub_I_value_semantic.cpp (value_semantic.cpp:67)
==14330==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14330==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14330==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14330== 
==14330== 21 bytes in 1 blocks are possibly lost in loss record 14 of 42
==14330==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14330==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14330==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14330==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14330==    by 0x4094367: _GLOBAL__sub_I_log.cpp (log.cpp:140)
==14330==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14330==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14330==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14330== 
==14330== 24 bytes in 1 blocks are definitely lost in loss record 28 of 42
==14330==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14330==    by 0x4123CA6: mongo::BackgroundJob::BackgroundJob(bool) (mutex.h:79)
==14330==    by 0x4092B02: _GLOBAL__sub_I_dbclient_rs.cpp (dbclient_rs.cpp:68)
==14330==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14330==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14330==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14330== 
==14330== 28 bytes in 1 blocks are possibly lost in loss record 30 of 42
==14330==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14330==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14330==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14330==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14330==    by 0x409439D: _GLOBAL__sub_I_log.cpp (log.cpp:141)
==14330==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14330==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14330==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14330== 
==14330== LEAK SUMMARY:
==14330==    definitely lost: 40 bytes in 2 blocks
==14330==    indirectly lost: 0 bytes in 0 blocks
==14330==      possibly lost: 49 bytes in 2 blocks
==14330==    still reachable: 1,049,780 bytes in 39 blocks
==14330==         suppressed: 0 bytes in 0 blocks
==14330== Reachable blocks (those to which a pointer was found) are not shown.
==14330== To see them, rerun with: --leak-check=full --show-reachable=yes
==14330== 
==14330== For counts of detected and suppressed errors, rerun with: -v
==14330== ERROR SUMMARY: 8 errors from 6 contexts (suppressed: 30 from 10)

Comment by Andrew Morrow (Inactive) [ 01/Oct/13 ]

Actually, never mind. I just repro'ed locally.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Sure. Can you send me your ssh public key?

Comment by Andrew Morrow (Inactive) [ 01/Oct/13 ]

adinoyi.omuya@10gen.com Can you re-run the above with --track-origins=yes (or show me how to log into the box and do it myself). I tried to repro this locally but was unable to do so so I'd like to try on the system itself.

Comment by Ian Whalen (Inactive) [ 01/Oct/13 ]

sidenote: not sure if you're already doing this, but whatever changes we're making to installed packages, output logging, etc. need to make their way onto the builders so that we don't need to go through this manual process again.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Using valgrind:

[ec2-user@ip-10-165-38-116 artifacts]$ valgrind --tool=memcheck --leak-check=yes  ./sharedclient/secondExample --port 27999
==14312== Memcheck, a memory error detector
==14312== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==14312== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==14312== Command: ./sharedclient/secondExample --port 27999
==14312== 
 
using cursor
	{ "_id" : { "$oid" : "524b1890d57683f19fd004e4" }, "name" : "eliot", "num" : 17 }
	{ "_id" : { "$oid" : "524b1890d57683f19fd004e5" }, "name" : "sara", "num" : 24 }
==14312== Invalid read of size 4
==14312==    at 0x42B1AB3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (atomicity.h:48)
==14312==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14312==    by 0x43811F6: __cxa_finalize (cxa_finalize.c:56)
==14312==    by 0x4095913: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14312==    by 0x41A2488: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14312==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14312==    by 0x4380EEC: exit (exit.c:100)
==14312==    by 0x4368A0A: (below main) (libc-start.c:258)
==14312==  Address 0x45f6740 is 8 bytes inside a block of size 16 free'd
==14312==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14312==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14312==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14312==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14312==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14312==    by 0x4380EEC: exit (exit.c:100)
==14312==    by 0x4368A0A: (below main) (libc-start.c:258)
==14312== 
==14312== Invalid free() / delete / delete[] / realloc()
==14312==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14312==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14312==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14312==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14312==    by 0x43811F6: __cxa_finalize (cxa_finalize.c:56)
==14312==    by 0x4095913: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14312==    by 0x41A2488: ??? (in /home/ec2-user/artifacts/sharedclient/libmongoclient.so)
==14312==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14312==    by 0x4380EEC: exit (exit.c:100)
==14312==    by 0x4368A0A: (below main) (libc-start.c:258)
==14312==  Address 0x45f6738 is 0 bytes inside a block of size 16 free'd
==14312==    at 0x4025892: operator delete(void*) (vg_replace_malloc.c:480)
==14312==    by 0x42B1A8E: std::string::_Rep::_M_destroy(std::allocator<char> const&) (new_allocator.h:100)
==14312==    by 0x42B1AD3: std::string::_Rep::_M_dispose(std::allocator<char> const&) [clone .part.13] (basic_string.h:246)
==14312==    by 0x42B1B29: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() (basic_string.h:536)
==14312==    by 0x4380E60: __run_exit_handlers (exit.c:78)
==14312==    by 0x4380EEC: exit (exit.c:100)
==14312==    by 0x4368A0A: (below main) (libc-start.c:258)
==14312== 
==14312== 
==14312== HEAP SUMMARY:
==14312==     in use at exit: 1,049,869 bytes in 43 blocks
==14312==   total heap usage: 461 allocs, 419 frees, 1,138,975 bytes allocated
==14312== 
==14312== 16 bytes in 1 blocks are definitely lost in loss record 13 of 42
==14312==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14312==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14312==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14312==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14312==    by 0x409573A: _GLOBAL__sub_I_value_semantic.cpp (value_semantic.cpp:67)
==14312==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14312==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14312==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14312== 
==14312== 21 bytes in 1 blocks are possibly lost in loss record 14 of 42
==14312==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14312==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14312==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14312==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14312==    by 0x4094367: _GLOBAL__sub_I_log.cpp (log.cpp:140)
==14312==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14312==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14312==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14312== 
==14312== 24 bytes in 1 blocks are definitely lost in loss record 28 of 42
==14312==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14312==    by 0x4123CA6: mongo::BackgroundJob::BackgroundJob(bool) (mutex.h:79)
==14312==    by 0x4092B02: _GLOBAL__sub_I_dbclient_rs.cpp (dbclient_rs.cpp:68)
==14312==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14312==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14312==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14312== 
==14312== 28 bytes in 1 blocks are possibly lost in loss record 30 of 42
==14312==    at 0x40267AB: operator new(unsigned int) (vg_replace_malloc.c:292)
==14312==    by 0x42B1925: std::string::_Rep::_S_create(unsigned int, unsigned int, std::allocator<char> const&) (new_allocator.h:94)
==14312==    by 0x42B35B7: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:140)
==14312==    by 0x42B3C40: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.h:1722)
==14312==    by 0x409439D: _GLOBAL__sub_I_log.cpp (log.cpp:141)
==14312==    by 0x400E801: call_init.part.0 (dl-init.c:85)
==14312==    by 0x400E8E7: _dl_init (dl-init.c:52)
==14312==    by 0x400114E: ??? (in /lib/ld-2.12.so)
==14312== 
==14312== LEAK SUMMARY:
==14312==    definitely lost: 40 bytes in 2 blocks
==14312==    indirectly lost: 0 bytes in 0 blocks
==14312==      possibly lost: 49 bytes in 2 blocks
==14312==    still reachable: 1,049,780 bytes in 39 blocks
==14312==         suppressed: 0 bytes in 0 blocks
==14312== Reachable blocks (those to which a pointer was found) are not shown.
==14312== To see them, rerun with: --leak-check=full --show-reachable=yes
==14312== 
==14312== For counts of detected and suppressed errors, rerun with: -v
==14312== ERROR SUMMARY: 8 errors from 6 contexts (suppressed: 30 from 10)

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Running with debug pkgs:

(gdb) run
Starting program: /home/ec2-user/artifacts/sharedclient/secondExample --port 27999
[Thread debugging using libthread_db enabled]
[New Thread 0xb7a22b70 (LWP 14219)]
[Thread 0xb7a22b70 (LWP 14219) exited]
using cursor
	{ "_id" : { "$oid" : "524b174dd57683f19fd004d8" }, "name" : "eliot", "num" : 17 }
	{ "_id" : { "$oid" : "524b174dd57683f19fd004d9" }, "name" : "sara", "num" : 24 }
 
Program received signal SIGSEGV, Segmentation fault.
malloc_consolidate (av=0xb7cba3e0) at malloc.c:5175
5175		  nextsize = chunksize(nextchunk);

Comment by Eric Milkie [ 01/Oct/13 ]

You're just missing the debuginfo packages; you can install them with those suggested yum commands (use root) and those gdb warnings should go away. They aren't the cause of the crash.
It looks like malloc() is hitting a segfault. If you can run this with valgrind, it might give more information on how we're corrupting the heap.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Thanks, using gdb, it looks like it's segfaulting after not finding librpm.so.1.

Starting program: /home/ec2-user/artifacts/sharedclient/secondExample --port 27999
warning: Unable to open "librpm.so.1" (librpm.so.1: cannot open shared object file: No such file or directory), missing debuginfos notifications will not be displayed
Missing separate debuginfo for /lib/ld-linux.so.2
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/07/dc80eb9d32e8d8f8233d143291cc3561be8557.debug
Missing separate debuginfo for /lib/i686/nosegneg/librt.so.1
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/07/c1a1566181f2247e4aedee86641a18cefd258c.debug
Missing separate debuginfo for /usr/lib/libstdc++.so.6
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/8b/335a9f1355388b4d60ae5a5793c71a27015b24.debug
Missing separate debuginfo for /lib/i686/nosegneg/libm.so.6
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b0/94ff2a1ff48d48e8aa2544da0095dcefe38ff7.debug
Missing separate debuginfo for /lib/libgcc_s.so.1
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a8/bbc4f820a5d18e669bcf4e2bfead780327e691.debug
Missing separate debuginfo for /lib/i686/nosegneg/libpthread.so.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d1/f7251021d2a7b727a459a39f4d1705b1c54e03.debug
[Thread debugging using libthread_db enabled]
Missing separate debuginfo for /lib/i686/nosegneg/libc.so.6
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/cf/3db046095b644349672276acb0b08bbd9bbbb9.debug
[New Thread 0xb7a22b70 (LWP 14059)]
[Thread 0xb7a22b70 (LWP 14059) exited]
using cursor
	{ "_id" : { "$oid" : "524b114ad57683f19fd004c6" }, "name" : "eliot", "num" : 17 }
	{ "_id" : { "$oid" : "524b114ad57683f19fd004c7" }, "name" : "sara", "num" : 24 }
 
Program received signal SIGSEGV, Segmentation fault.
0xb7b9cfc2 in malloc_consolidate.part.3 () from /lib/i686/nosegneg/libc.so.6

Comment by Eric Milkie [ 01/Oct/13 ]

I would just run the binary using gdb (no python), and step through the code and see how it exits.

Comment by Adinoyi Omuya [ 01/Oct/13 ]

Ran this twice on a fresh linux-32 box and the same test failed. Buildlogger logs here and here. The first run had slightly more information than is available in MCI but the second run had even less. Both failed with a 245 exit status - similar to what's in the MCI logs.

I post more information once I'm able to run it using gdb.

Comment by Eric Milkie [ 30/Sep/13 ]

My changes were only to the build system and only affect Windows.
Since the commit that caused the failure is unknown, and the log isn't helpful, and buildbot is showing green, I recommend someone run the test by hand on the MCI builder, using gdb.

Generated at Thu Feb 08 03:24:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.