<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:21:41 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9869] C++ driver crash when mongodb is shutdowned</title>
                <link>https://jira.mongodb.org/browse/SERVER-9869</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;One of our applications crashed with the following stacktrace from the C++ mongodb driver thread:&lt;/p&gt;

&lt;p&gt;Program terminated with signal 6, Aborted.&lt;br/&gt;
#0  0x00007fe627b40945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64&lt;br/&gt;
64	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);&lt;br/&gt;
(gdb) bt&lt;br/&gt;
#0  0x00007fe627b40945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64&lt;br/&gt;
#1  0x00007fe627b4225b in abort () at abort.c:92&lt;br/&gt;
#2  0x00007fe627b3937e in __assert_fail_base (fmt=&amp;lt;value optimized out&amp;gt;, assertion=0x7fe62af586c4 &quot;!pthread_mutex_lock(&amp;amp;m)&quot;, &lt;br/&gt;
    file=0x7fe62af58790 &quot;/usr/include/boost/thread/pthread/mutex.hpp&quot;, line=&amp;lt;value optimized out&amp;gt;, function=&amp;lt;value optimized out&amp;gt;) at assert.c:96&lt;br/&gt;
#3  0x00007fe627b39422 in __assert_fail (assertion=0x7fe62af586c4 &quot;!pthread_mutex_lock(&amp;amp;m)&quot;, file=0x7fe62af58790 &quot;/usr/include/boost/thread/pthread/mutex.hpp&quot;, line=125, &lt;br/&gt;
    function=0x7fe62af65100 &quot;void boost::timed_mutex::lock()&quot;) at assert.c:105&lt;br/&gt;
#4  0x00007fe62af3f2c3 in lock (this=0x2158180, t=0x0) at /usr/include/boost/thread/pthread/mutex.hpp:125&lt;br/&gt;
#5  lock (this=0x2158180, t=0x0) at /usr/include/boost/thread/locks.hpp:349&lt;br/&gt;
#6  unique_lock (this=0x2158180, t=0x0) at /usr/include/boost/thread/locks.hpp:227&lt;br/&gt;
#7  scoped_lock (this=0x2158180, t=0x0) at src/mongo/util/concurrency/mutex.h:104&lt;br/&gt;
#8  mongo::Logstream::flush (this=0x2158180, t=0x0) at src/mongo/util/log.cpp:357&lt;br/&gt;
#9  0x00007fe62af3fb8e in mongo::Logstream::operator&amp;lt;&amp;lt; (this=&amp;lt;value optimized out&amp;gt;, _endl=&amp;lt;value optimized out&amp;gt;) at src/mongo/util/log.h:307&lt;br/&gt;
#10 0x00007fe62aecb128 in mongo::ReplicaSetMonitor::_checkConnection (this=0x1f49920, conn=0x1f94c80, maybePrimary=&amp;lt;value optimized out&amp;gt;, verbose=true, nodesOffset=0)&lt;br/&gt;
    at src/mongo/client/dbclient_rs.cpp:766&lt;br/&gt;
#11 0x00007fe62aecb66c in mongo::ReplicaSetMonitor::_check (this=0x1f49920, checkAllSecondaries=true) at src/mongo/client/dbclient_rs.cpp:814&lt;br/&gt;
#12 0x00007fe62aecf54c in mongo::ReplicaSetMonitor::check (this=0x1f49920, checkAllSecondaries=true) at src/mongo/client/dbclient_rs.cpp:955&lt;br/&gt;
#13 0x00007fe62aecf7a3 in mongo::ReplicaSetMonitor::checkAll (checkAllSecondaries=true) at src/mongo/client/dbclient_rs.cpp:322&lt;br/&gt;
#14 0x00007fe62aed7f74 in mongo::ReplicaSetMonitorWatcher::run (this=&amp;lt;value optimized out&amp;gt;) at src/mongo/client/dbclient_rs.cpp:202&lt;br/&gt;
#15 0x00007fe62af2c3db in mongo::BackgroundJob::jobBody (this=0x7fe62b196000, status=...) at src/mongo/util/background.cpp:64&lt;br/&gt;
#16 0x00007fe62af2e309 in operator() (this=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/bind/mem_fn_template.hpp:162&lt;br/&gt;
#17 operator()&amp;lt;boost::_mfi::mf1&amp;lt;void, mongo::BackgroundJob, boost::shared_ptr&amp;lt;mongo::BackgroundJob::JobStatus&amp;gt; &amp;gt;, boost::_bi::list0&amp;gt; (this=&amp;lt;value optimized out&amp;gt;)&lt;br/&gt;
    at /usr/include/boost/bind/bind.hpp:306&lt;br/&gt;
#18 operator() (this=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/bind/bind_template.hpp:20&lt;br/&gt;
#19 boost::detail::thread_data&amp;lt;boost::_bi::bind_t&amp;lt;void, boost::_mfi::mf1&amp;lt;void, mongo::BackgroundJob, boost::shared_ptr&amp;lt;mongo::BackgroundJob::JobStatus&amp;gt; &amp;gt;, boost::_bi::list2&amp;lt;boost::_bi::value&amp;lt;mongo::BackgroundJob*&amp;gt;, boost::_bi::value&amp;lt;boost::shared_ptr&amp;lt;mongo::BackgroundJob::JobStatus&amp;gt; &amp;gt; &amp;gt; &amp;gt; &amp;gt;::run (this=&amp;lt;value optimized out&amp;gt;)&lt;br/&gt;
    at /usr/include/boost/thread/detail/thread.hpp:56&lt;br/&gt;
#20 0x00007fe62885fd97 in boost::(anonymous namespace)::thread_proxy (param=&amp;lt;value optimized out&amp;gt;) at /usr/src/debug/boost-1.41.0.cmake0/libs/thread/src/pthread/thread.cpp:120&lt;br/&gt;
#21 0x00007fe628640c6b in start_thread (arg=0x7fe620d04700) at pthread_create.c:301&lt;br/&gt;
#22 0x00007fe627bee5ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115&lt;br/&gt;
(gdb)&lt;/p&gt;

&lt;p&gt;It seems that the crash happened when mongodb was shutdowned. There is a lot of inactivity in mongodb log until the moment of the shutdown:&lt;br/&gt;
Sat May 11 10:00:42 got signal 15 (Terminated), will terminate after current cmd ends&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; now exiting&lt;br/&gt;
Sat May 11 10:00:42 dbexit: &lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: going to close listening sockets...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; closing listening socket: 6&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; closing listening socket: 7&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; removing socket file: /tmp/mongodb-27017.sock&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21&amp;#93;&lt;/span&gt; command vmdb.$cmd command: &lt;/p&gt;
{ getlasterror: 1, w: 2, wtimeout: 0 }
&lt;p&gt; ntoreturn:1 keyUpdates:0  reslen:135 85934181ms&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn31&amp;#93;&lt;/span&gt; command vmdb.$cmd command: &lt;/p&gt;
{ getlasterror: 1, w: 2, wtimeout: 0 }
&lt;p&gt; ntoreturn:1 keyUpdates:0  reslen:135 85812168ms&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn21&amp;#93;&lt;/span&gt; SocketException handling request, closing client connection: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;2&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;172.31.1.9:48545&amp;#93;&lt;/span&gt; &lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: going to flush diaglog...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: going to close sockets...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: waiting for fs preallocator...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: lock for final commit...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: final commit...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn31&amp;#93;&lt;/span&gt; SocketException handling request, closing client connection: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;2&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;172.31.1.9:48591&amp;#93;&lt;/span&gt; &lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: closing all files...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; closeAllFiles() finished&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; journalCleanup...&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; removeJournalFiles&lt;br/&gt;
Sat May 11 10:00:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;interruptThread&amp;#93;&lt;/span&gt; shutdown: removing fs lock...&lt;br/&gt;
Sat May 11 10:00:42 dbexit: really exiting now&lt;br/&gt;
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default&lt;br/&gt;
forked process: 1216&lt;br/&gt;
all output going to: /var/log/mongodb/mongodb.log&lt;br/&gt;
log file &lt;span class=&quot;error&quot;&gt;&amp;#91;/var/log/mongodb/mongodb.log&amp;#93;&lt;/span&gt; exists; copied to temporary file &lt;span class=&quot;error&quot;&gt;&amp;#91;/var/log/mongodb/mongodb.log.2013-05-11T10-01-39&amp;#93;&lt;/span&gt;&lt;br/&gt;
child process started successfully, parent exiting&lt;/p&gt;

&lt;p&gt;Mongodb has only one node:&lt;br/&gt;
mongo&lt;br/&gt;
MongoDB shell version: 2.2.3&lt;br/&gt;
connecting to: test&lt;br/&gt;
sipxecs:PRIMARY&amp;gt; rs.config()&lt;br/&gt;
{&lt;br/&gt;
        &quot;_id&quot; : &quot;sipxecs&quot;,&lt;br/&gt;
        &quot;version&quot; : 1,&lt;br/&gt;
        &quot;members&quot; : [&lt;/p&gt;
                {
                        &quot;_id&quot; : 0,
                        &quot;host&quot; : &quot;openuc.ezuce.com:27017&quot;,
                        &quot;priority&quot; : 2
                }
&lt;p&gt;        ]&lt;br/&gt;
}&lt;/p&gt;</description>
                <environment>Linux 3.4.37-40.44.amzn1.x86_64 #1 SMP Thu Mar 21 01:17:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux</environment>
        <key id="78106">SERVER-9869</key>
            <summary>C++ driver crash when mongodb is shutdowned</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="andrew.morrow@mongodb.com">Andrew Morrow</assignee>
                                    <reporter username="dtacalau">Daniel Stefan Tacalau</reporter>
                        <labels>
                    </labels>
                <created>Thu, 6 Jun 2013 13:48:26 +0000</created>
                <updated>Mon, 11 Jul 2016 17:39:32 +0000</updated>
                            <resolved>Wed, 26 Jun 2013 15:56:46 +0000</resolved>
                                    <version>2.2.3</version>
                                                    <component>Internal Client</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="368207" author="acm" created="Wed, 26 Jun 2013 15:56:46 +0000"  >&lt;p&gt;Closing since the solution of using dbexit appears to be working. Feel free to re-open if the issue recurs.&lt;/p&gt;</comment>
                            <comment id="363540" author="acm" created="Wed, 19 Jun 2013 18:01:26 +0000"  >&lt;p&gt;Hi Daniel -&lt;/p&gt;

&lt;p&gt;I&apos;ve verified with another engineer that it is OK for you to call mongo::dbexit as you are doing. Since you aren&apos;t able to reproduce the crash right now, and since you have opened another ticket for the logging issue (thanks for doing that), I think it makes sense to close this ticket for now. If you experience the original crash now that you are using mongo::dbexit, you can re-open this ticket and we can investigate further.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Andrew&lt;/p&gt;</comment>
                            <comment id="363497" author="dtacalau" created="Wed, 19 Jun 2013 17:05:42 +0000"  >&lt;p&gt;Andrew,&lt;/p&gt;

&lt;p&gt;Sorry for the confusing answer. Let me clarify:&lt;br/&gt;
1. When I said &quot;the one above&quot; I meant comment with date &quot;Jun 19 2013 10:04:32 AM UTC&quot;. &lt;br/&gt;
2. AFAIK mongo::dbexit cleared only &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8891&quot; title=&quot;Simple client fail with segmentation fault in mongoclient library&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-8891&quot;&gt;&lt;del&gt;SERVER-8891&lt;/del&gt;&lt;/a&gt; style crash, but I might be wrong. As I said it&apos;s not reproducible and I haven&apos;t seen it since I opened this ticket.&lt;br/&gt;
3. Short answer: yes.&lt;/p&gt;

&lt;p&gt;I&apos;ll open a separate ticket for the logging stuff.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Daniel&lt;/p&gt;</comment>
                            <comment id="363354" author="acm" created="Wed, 19 Jun 2013 14:11:09 +0000"  >&lt;p&gt;Hi Daniel -&lt;/p&gt;

&lt;p&gt;A few questions and a suggestion:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;When you say &quot;the one above&quot;, are you referring to the crash in your original report, or the one you posted, or the one from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8891&quot; title=&quot;Simple client fail with segmentation fault in mongoclient library&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-8891&quot;&gt;&lt;del&gt;SERVER-8891&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;When you add the call to mongo::dbexit, does this also clear up your original crash? Or only the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8891&quot; title=&quot;Simple client fail with segmentation fault in mongoclient library&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-8891&quot;&gt;&lt;del&gt;SERVER-8891&lt;/del&gt;&lt;/a&gt; style crash?&lt;/li&gt;
	&lt;li&gt;Assuming that dbexit is working for you, and that it is proper for you to call it (I&apos;m looking into that now) would that be a sufficient resolution for this issue for you?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;For the C++ driver logging questions, I&apos;d suggest either opening a separate ticket so we can track the issues independently.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Andrew&lt;/p&gt;

</comment>
                            <comment id="363222" author="dtacalau" created="Wed, 19 Jun 2013 10:10:37 +0000"  >&lt;p&gt;Andrew,&lt;/p&gt;

&lt;p&gt;I forgot to mention that I always get crashes on exit similar with the one above in most of our applications that are using the C++ mongo driver. After researching the matter I &lt;br/&gt;
found out to be a known problem with static constructors/destructors so I solved it by using a call to mongo::dbexit(mongo::EXIT_CLEAN). Is this OK?&lt;/p&gt;

&lt;p&gt;Is there any way I can enable in my app C++ driver logging on DEBUG level? I&apos;ve tried using mongo::initLogging(&amp;lt;some-file&amp;gt;, true) but it does not write too much. It seems to write only messages logged by calls to &quot;log()&quot; and it does not write messages logged by &quot;log(&amp;lt;some-level&amp;gt;)&quot;. &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Daniel&lt;/p&gt;</comment>
                            <comment id="363219" author="dtacalau" created="Wed, 19 Jun 2013 10:04:32 +0000"  >&lt;p&gt;Andrew,&lt;/p&gt;

&lt;p&gt;Here is what I get when I run the test program from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8891&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-8891&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;sh-4.1# gdb --args ./a.out &lt;br/&gt;
GNU gdb (GDB) Amazon Linux (7.2-60.13.amzn1)&lt;br/&gt;
Copyright (C) 2010 Free Software Foundation, Inc.&lt;br/&gt;
License GPLv3+: GNU GPL version 3 or later &amp;lt;&lt;a href=&quot;http://gnu.org/licenses/gpl.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://gnu.org/licenses/gpl.html&lt;/a&gt;&amp;gt;&lt;br/&gt;
This is free software: you are free to change and redistribute it.&lt;br/&gt;
There is NO WARRANTY, to the extent permitted by law.  Type &quot;show copying&quot;&lt;br/&gt;
and &quot;show warranty&quot; for details.&lt;br/&gt;
This GDB was configured as &quot;x86_64-amazon-linux-gnu&quot;.&lt;br/&gt;
For bug reporting instructions, please see:&lt;br/&gt;
&amp;lt;&lt;a href=&quot;http://www.gnu.org/software/gdb/bugs/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.gnu.org/software/gdb/bugs/&lt;/a&gt;&amp;gt;...&lt;br/&gt;
Reading symbols from /home/ec2-user/a.out...done.&lt;br/&gt;
(gdb) r&lt;br/&gt;
Starting program: /home/ec2-user/a.out &lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;Thread debugging using libthread_db enabled&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jun 19 10:01:17 starting new replica set monitor for replica set c2 with seed of localhost:27017&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;New Thread 0x7ffff46ac700 (LWP 12811)&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jun 19 10:01:17 successfully connected to seed localhost:27017 for replica set c2&lt;br/&gt;
Wed Jun 19 10:01:17 EINTR retry 1&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;Thread 0x7ffff46ac700 (LWP 12811) exited&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jun 19 10:01:17 warning: node: localhost:27017 isn&apos;t a part of set: c2 ismaster: &lt;/p&gt;
{ setName: &quot;sipxecs&quot;, ismaster: true, secondary: false, hosts: [ &quot;openuc.ezuce.com:27017&quot; ], primary: &quot;openuc.ezuce.com:27017&quot;, me: &quot;openuc.ezuce.com:27017&quot;, maxBsonObjectSize: 16777216, localTime: new Date(1371636077662), ok: 1.0 }
&lt;p&gt;Wed Jun 19 10:01:19 warning: No primary detected for set c2&lt;br/&gt;
Wed Jun 19 10:01:19 All nodes for set c2 are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks&lt;br/&gt;
Wed Jun 19 10:01:19 replica set monitor for replica set c2 started, address is c2/&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;New Thread 0x7ffff46ac700 (LWP 12815)&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Jun 19 10:01:19 deleting replica set monitor for: c2/&lt;/p&gt;

&lt;p&gt;Program received signal SIGSEGV, Segmentation fault.&lt;br/&gt;
std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt;::compare (this=Unhandled dwarf expression opcode 0xf3&lt;br/&gt;
) at /usr/src/debug/gcc-4.7.2-20121109/obj-x86_64-amazon-linux/x86_64-amazon-linux/libstdc++-v3/include/bits/basic_string.h:2224&lt;br/&gt;
2224            const size_type __size = this-&amp;gt;size();&lt;br/&gt;
(gdb) bt&lt;br/&gt;
#0  std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt;::compare (this=Unhandled dwarf expression opcode 0xf3&lt;br/&gt;
) at /usr/src/debug/gcc-4.7.2-20121109/obj-x86_64-amazon-linux/x86_64-amazon-linux/libstdc++-v3/include/bits/basic_string.h:2224&lt;br/&gt;
#1  0x00007ffff75f7c3d in operator&amp;lt; &amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; (this=0x7ffff78b9080, __k=&quot;c2&quot;) at /usr/include/c++/4.4.7/bits/basic_string.h:2317&lt;br/&gt;
#2  operator() (this=0x7ffff78b9080, __k=&quot;c2&quot;) at /usr/include/c++/4.4.7/bits/stl_function.h:230&lt;br/&gt;
#3  _M_lower_bound (this=0x7ffff78b9080, __k=&quot;c2&quot;) at /usr/include/c++/4.4.7/bits/stl_tree.h:986&lt;br/&gt;
#4  lower_bound (this=0x7ffff78b9080, __k=&quot;c2&quot;) at /usr/include/c++/4.4.7/bits/stl_tree.h:745&lt;br/&gt;
#5  lower_bound (this=0x7ffff78b9080, __k=&quot;c2&quot;) at /usr/include/c++/4.4.7/bits/stl_map.h:701&lt;br/&gt;
#6  std::map&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt;, std::vector&amp;lt;mongo::HostAndPort, std::allocator&amp;lt;mongo::HostAndPort&amp;gt; &amp;gt;, std::less&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; &amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; const, std::vector&amp;lt;mongo::HostAndPort, std::allocator&amp;lt;mongo::HostAndPort&amp;gt; &amp;gt; &amp;gt; &amp;gt; &amp;gt;::operator[] (this=0x7ffff78b9080, __k=&quot;c2&quot;)&lt;br/&gt;
    at /usr/include/c++/4.4.7/bits/stl_map.h:447&lt;br/&gt;
#7  0x00007ffff75e7e5b in mongo::ReplicaSetMonitor::_cacheServerAddresses_inlock (this=0x612280) at src/mongo/client/dbclient_rs.cpp:265&lt;br/&gt;
#8  0x00007ffff75e9885 in mongo::ReplicaSetMonitor::~ReplicaSetMonitor (this=0x612280, __in_chrg=&amp;lt;value optimized out&amp;gt;) at src/mongo/client/dbclient_rs.cpp:257&lt;br/&gt;
#9  0x00007ffff75fa3e2 in checked_delete&amp;lt;mongo::ReplicaSetMonitor&amp;gt; (this=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/checked_delete.hpp:34&lt;br/&gt;
#10 boost::detail::sp_counted_impl_p&amp;lt;mongo::ReplicaSetMonitor&amp;gt;::dispose (this=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78&lt;br/&gt;
#11 0x00007ffff75f4ff9 in release (this=&amp;lt;value optimized out&amp;gt;, __in_chrg=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145&lt;br/&gt;
#12 boost::detail::shared_count::~shared_count (this=&amp;lt;value optimized out&amp;gt;, __in_chrg=&amp;lt;value optimized out&amp;gt;) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217&lt;br/&gt;
#13 0x00007ffff75f875e in ~shared_ptr (this=0x7ffff78b9040, __x=0x612240) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169&lt;br/&gt;
#14 ~pair (this=0x7ffff78b9040, __x=0x612240) at /usr/include/c++/4.4.7/bits/stl_pair.h:68&lt;br/&gt;
#15 destroy (this=0x7ffff78b9040, __x=0x612240) at /usr/include/c++/4.4.7/ext/new_allocator.h:115&lt;br/&gt;
#16 _M_destroy_node (this=0x7ffff78b9040, __x=0x612240) at /usr/include/c++/4.4.7/bits/stl_tree.h:383&lt;br/&gt;
#17 std::_Rb_tree&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt;, std::pair&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt;, std::_Select1st&amp;lt;std::pair&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt;, std::less&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; &amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::_M_erase (this=0x7ffff78b9040, __x=0x612240) at /usr/include/c++/4.4.7/bits/stl_tree.h:972&lt;br/&gt;
#18 0x00007ffff75f888d in ~_Rb_tree (this=&amp;lt;value optimized out&amp;gt;, __in_chrg=&amp;lt;value optimized out&amp;gt;) at /usr/include/c++/4.4.7/bits/stl_tree.h:614&lt;br/&gt;
#19 std::map&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt;, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt;, std::less&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; &amp;gt;, std::allocator&amp;lt;std::pair&amp;lt;std::basic_string&amp;lt;char, std::char_traits&amp;lt;char&amp;gt;, std::allocator&amp;lt;char&amp;gt; &amp;gt; const, boost::shared_ptr&amp;lt;mongo::ReplicaSetMonitor&amp;gt; &amp;gt; &amp;gt; &amp;gt;::~map (this=&amp;lt;value optimized out&amp;gt;, __in_chrg=&amp;lt;value optimized out&amp;gt;)&lt;br/&gt;
    at /usr/include/c++/4.4.7/bits/stl_map.h:87&lt;br/&gt;
#20 0x00007ffff6d6a38d in __cxa_finalize (d=0x7ffff78b11c0) at cxa_finalize.c:56&lt;br/&gt;
#21 0x00007ffff75be8c6 in __do_global_dtors_aux () from /usr/lib64/libmongoclient.so&lt;br/&gt;
#22 0x00007fffffffe4f0 in ?? ()&lt;br/&gt;
#23 0x00007fffffffe630 in ?? ()&lt;br/&gt;
#24 0x00007ffff767af21 in _fini () from /usr/lib64/libmongoclient.so&lt;br/&gt;
#25 0x00007fffffffe630 in ?? ()&lt;br/&gt;
#26 0x00007ffff7ded31d in _dl_fini () at dl-fini.c:259&lt;br/&gt;
Backtrace stopped: previous frame inner to this frame (corrupt stack?)&lt;/p&gt;

&lt;p&gt;Daniel&lt;/p&gt;</comment>
                            <comment id="362757" author="acm" created="Tue, 18 Jun 2013 17:42:19 +0000"  >
&lt;p&gt;Hi Daniel -&lt;/p&gt;

&lt;p&gt;I wonder if you may be affected by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-8891&quot; title=&quot;Simple client fail with segmentation fault in mongoclient library&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-8891&quot;&gt;&lt;del&gt;SERVER-8891&lt;/del&gt;&lt;/a&gt;. Would you be able to test the little example program at the beginning of that ticket in your environment and see if you observe similar behavior? The stack traces aren&apos;t a perfect match between that ticket and this one, but there is enough similarity that I&apos;m curious.&lt;/p&gt;

&lt;p&gt;As to your question about instrumentation, one possibility would be to use valgrind to see if there is a static destructor ordering problem, which would almost certainly show up as a use after free, or some other sort of similar memory error.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Andrew&lt;/p&gt;</comment>
                            <comment id="362643" author="dtacalau" created="Tue, 18 Jun 2013 15:20:49 +0000"  >&lt;p&gt;Andrew,&lt;/p&gt;

&lt;p&gt;This hasn&apos;t reproduced so far. &lt;/p&gt;

&lt;p&gt;1. We&apos;re using C++ driver 2.2.3 from CentOS 6.4 repo.&lt;br/&gt;
2. Boost version is 1.41 .&lt;br/&gt;
3. The crash occurs only sometimes.&lt;br/&gt;
4. I don&apos;t have other clients logs.&lt;/p&gt;

&lt;p&gt;Is there any kind of instrumentation that I can enable in the C++ driver so that when the crash occur again I&apos;ll have some reports?&lt;/p&gt;

&lt;p&gt;Thanks for looking into this,&lt;br/&gt;
Daniel&lt;/p&gt;
</comment>
                            <comment id="361907" author="acm" created="Mon, 17 Jun 2013 16:44:29 +0000"  >&lt;p&gt;Hi Daniel -&lt;/p&gt;

&lt;p&gt;Just checking in to see if this is still an issue for you.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Andrew&lt;/p&gt;</comment>
                            <comment id="358663" author="acm" created="Wed, 12 Jun 2013 15:08:08 +0000"  >&lt;p&gt;Hi Daniel - &lt;/p&gt;

&lt;p&gt;Could you provide some additional details to help us diagnose the issue?&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Did you build the driver from the C++ driver source tarballs? If so, what version? If not, how did you obtain the driver library?&lt;/li&gt;
	&lt;li&gt;The driver appears to be built against boost from /usr/include. What version of boost is in place on the system?&lt;/li&gt;
	&lt;li&gt;Is this crash consistently reproducible when you terminate the server, or occurs only sometimes?&lt;/li&gt;
	&lt;li&gt;If you can reproduce this consistently, does the client program produce any logging output in the time leading up to the crash? If not, do you happen to have archived client logs from when this crash occurred?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Andrew&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 12 Jun 2013 15:08:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 34 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 34 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>dtacalau</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmqvj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrr06n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>71384</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsuzkf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>