[SERVER-4576] mongos stack traces after bind failure Created: 28/Dec/11  Updated: 08/Mar/13  Resolved: 27/Aug/12

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

This happened while running sharding/auth.js on my machine with --dd and --durableDefaultOff options

 
shell: started program /Users/aaron/src/mongo/mongos --port 31000 --configdb Aaron-Staples-MacBook-Pro.local:40000 --keyFile jstests/libs/key1 --chunkSize 1
 m31000| Wed Dec 28 14:21:13 Warning: soft rlimits too low. Number of files is 256, should be at least 1000
 m31000| Wed Dec 28 14:21:13 /Users/aaron/src/mongo/mongos db version v2.1.0-pre-, pdfile version 4.5 starting (--help for usage)
 m31000| Wed Dec 28 14:21:13 git version: ff70c7c017840ec54897085e7d0bc5d4f8f5acb7
 m31000| Wed Dec 28 14:21:13 build info: Darwin Aaron-Staples-MacBook-Pro.local 11.0.0 Darwin Kernel Version 11.0.0: Sat Jun 18 12:56:35 PDT 2011; root:xnu-1699.22.73~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_47
 m31000| Wed Dec 28 14:21:13 [websvr] ERROR: listen(): bind() failed errno:48 Address already in use for socket: 0.0.0.0:32000
 m31000| Wed Dec 28 14:21:13 [websvr] ERROR:   addr already in use
 m31000| Wed Dec 28 14:21:13 [Balancer] about to contact config servers and shards
 m31000| Wed Dec 28 14:21:13 [Balancer] config servers and shards contacted successfully
 m31000| Wed Dec 28 14:21:13 [Balancer] balancer id: Aaron-Staples-MacBook-Pro.local:31000 started at Dec 28 14:21:13
 m31000| Wed Dec 28 14:21:13 [Balancer] created new distributed lock for balancer on Aaron-Staples-MacBook-Pro.local:40000 ( lock timeout : 900000, ping interval : 30000, process : 0 )
 m31000| Wed Dec 28 14:21:13 [Balancer] creating WriteBackListener for: Aaron-Staples-MacBook-Pro.local:40000 serverID: 4efb9659f52096cef2142469
 m31000| Wed Dec 28 14:21:13 [mongosMain] ERROR: listen(): bind() failed errno:48 Address already in use for socket: 0.0.0.0:31000
 m31000| Wed Dec 28 14:21:13 [mongosMain] ERROR:   addr already in use
 m31000| Wed Dec 28 14:21:13 [mongosMain] dbexit:  rc:0 
 m31000| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m31000| 0x105b5a005 0x105b746df 0x105b52b28 0x105b9af14 0x105b962b7 0x105bc6c5a 0x105bc6c7b 0x105bc9af3 0x105ba3523 0x105badb0d 0x105badbbe 0x105badc0c 0x105badcca 0x105badd09 0x105badd8c 0x105ba5c26 0x105ba61fb 0x7fff8b35b7c8 0x7fff8b35b652 0x105fce45c 
 m31000| Received signal 11
 m31000| Backtrace: 0x105b873bd 0x105b874ca 0x7fff8b3bccfa 0x7ff29840f7b8 0x105f48b42 0x105ba346b 0x105ba40cd 0x105ba6e86 0x105c35c2d 0x105c314dc 0x105c31a2f 0x105ef62b3 0x105fe820f 0x105b4c3cd 0x105b50b83 0x105b50c82 0x105b50d54 0x105b50d7f 0x106bde460 0x7fff8b3688bf 
 m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52495 #10 (10 connections now open)
 m40000| Wed Dec 28 14:21:13 [conn10]  authenticate: { authenticate: 1, nonce: "262cc0c05fd82b93", user: "__system", key: "38dbee07e6ce8ed25ab02a171e0608ec" }
 m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52496 #11 (11 connections now open)
 m40000| Wed Dec 28 14:21:13 [conn11]  authenticate: { authenticate: 1, nonce: "2c00d67641d56c51", user: "__system", key: "7460408a1c2d1cb0c6561c15ae088e15" }
 m40000| Wed Dec 28 14:21:13 [initandlisten] connection accepted from 10.1.10.180:52497 #12 (12 connections now open)
 m40000| Wed Dec 28 14:21:13 [conn12]  authenticate: { authenticate: 1, nonce: "50b752ae077a597f", user: "__system", key: "8758e26fb74e1733d245735c34bd15ea" }
 m40000| Wed Dec 28 14:21:13 [conn10] end connection 10.1.10.180:52495 (11 connections now open)
 m40000| Wed Dec 28 14:21:13 [conn10] thread conn stack usage was 15256 bytes
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 m31000| WARNING: mongod wrote null bytes to output
 
 m31000| 0   mongos                              0x0000000105b873bd _ZN5mongoL18formattedBacktraceEi + 451   mongos                              0x0000000105b874ca _ZN5mongo17printStackAndExitEi + 902   libsystem_c.dylib                   0x00007fff8b3bccfa _sigtramp + 263   ???                                 0x00007ff29840f7b8 0x0 + 1406799132077364   mongos                              0x0000000105f48b42 _ZN5mongo22ShardingConnectionHook8onCreateEPNS_12DBClientBaseE + 11865   mongos                              0x0000000105ba346b _ZN5mongo16DBConnectionPool8onCreateEPNS_12DBClientBaseE + 1356   mongos                              0x0000000105ba40cd _ZN5mongo16DBConnectionPool13_finishCreateERKSsdPNS_12DBClientBaseE + 3957   mongos                              0x0000000105ba6e86 _ZN5mongo16DBConnectionPool3getERKSsd + 19528   mongos                              0x0000000105c35c2d _ZN5mongo17ClientConnections3getERKSsS2_ + 3799   mongos                              0x0000000105c314dc _ZN5mongo15ShardConnection5_initEv + 15210  mongos                              0x0000000105c31a2f _ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSsN5boost10shared_ptrIKNS_12ChunkManagerEEE + 17711  mongos                              0x0000000105ef62b3 _ZNK5mongo4Grid13shouldBalanceEv + 187 0   mongos                              0x0000000105b5a005 _ZN5mongo15printStackTraceERSo + 37
 m31000|  1   mongos                              0x0000000105b746df _ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev + 87
 m31000|  2   mongos                              0x0000000105b52b28 _ZN5mongo5mutex11scoped_lockC1ERS0_ + 40
 m31000|  3   mongos                              0x0000000105b9af14 _ZN5mongo5Ports5eraseEPNS_13MessagingPortE + 48
 m31000|  4   mongos                              0x0000000105b962b7 _ZN5mongo13MessagingPortD0Ev + 143
 m31000|  5   mongos                              0x0000000105bc6c5a _ZN5boost14checked_deleteIN5mongo13MessagingPortEEEvPT_ + 42
 m31000|  6   mongos                              0x0000000105bc6c7b _ZN5boost10scoped_ptrIN5mongo13MessagingPortEED1Ev + 27
 m31000|  7   mongos                              0x0000000105bc9af3 _ZN5mongo18DBClientConnectionD0Ev + 519
 m31000|  8   mongos                              0x0000000105ba3523 _ZN5mongo11PoolForHostD1Ev + 85
 m31000|  9   mongos                              0x0000000105badb0d _ZNSt4pairIKN5mongo16DBConnectionPool7PoolKeyENS0_11PoolForHostEED1Ev + 37
 m31000|  10  mongos                              0x0000000105badbbe _ZN9__gnu_cxx13new_allocatorISt4pairIKN5mongo16DBConnectionPool7PoolKeyENS2_11PoolForHostEEE7destroyEPS7_ + 28
 m31000|  11  mongos                              0x0000000105badc0c _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EE15_M_destroy_nodeEPSt13_Rb_tree_nodeIS6_E + 72
 m31000|  12  mongos                              0x0000000105badcca _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EE8_M_eraseEPSt13_Rb_tree_nodeIS6_E + 80
 m31000|  13  mongos                              0x0000000105badd09 _ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EED1Ev + 39
 m31000|  14  mongos                              0x0000000105badd8c _ZNSt3mapIN5mongo16DBConnectionPool7PoolKeyENS0_11PoolForHostENS1_14poolKeyCompareESaISt4pairIKS2_S3_EEED1Ev + 24
 m31000|  15  mongos                              0x0000000105ba5c26 _ZN5mongo16DBConnectionPoolD1Ev + 64
 m31000|  16  mongos                              0x0000000105ba61fb __tcf_6 + 27
 m31000|  17  libsystem_c.dylib                   0x00007fff8b35b7c8 __cxa_finalize + 274
 m31000|  18  libsystem_c.dylib                   0x00007fff8b35b652 exit + 18
 m31000|  19  mongos                              0x0000000105fce45c _ZN5mongo6dbexitENS_8ExitCodeEPKcb + 316

Seems like there is an issue relating to locking a mutex during destruction of a connection pool, and also potentially another issue in the sharding connection hook's onCreate(). And also it looks like there is a problem relating to printing two stack traces at the same time. But I think maybe stack printing should be pretty light weight don't know if anything can/should be done about that.



 Comments   
Comment by Spencer Brody (Inactive) [ 27/Aug/12 ]

I was able to reproduce the stack trace by running sharding/auth.js with mongods already running on ports 31000 and 32000 to force a bind failure, when using the git version shown above. On master, it did not show up, so it seems like the problem has gone away.

Comment by Aaron Staple [ 29/Dec/11 ]

Also, in some other testing without a bind failure:

 m30999| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m30999| 0x7e1bdf 0x7f1059 0x7636bf 0x804ad8 0x802e96 0x802f1e 0x821698 0x81fda1 0x81e0cd 0x81e18c 0x8302b0 0x83299c 0x760fe6 0x761075 0x82cf68 0x82dd3d 0x83215c 0x831768 0x830537 0x82f2b7 
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo15printStackTraceERSo+0x27) [0x7e1bdf]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev+0x51) [0x7f1059]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo5mutex11scoped_lockC2ERS0_+0x29) [0x7636bf]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo5Ports5eraseEPNS_13MessagingPortE+0x28) [0x804ad8]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo13MessagingPortD1Ev+0x74) [0x802e96]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo13MessagingPortD0Ev+0x18) [0x802f1e]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost14checked_deleteIN5mongo13MessagingPortEEEvPT_+0x2a) [0x821698]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost10scoped_ptrIN5mongo13MessagingPortEED1Ev+0x1b) [0x81fda1]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo18DBClientConnectionD2Ev+0x9d) [0x81e0cd]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo18DBClientConnectionD0Ev+0x18) [0x81e18c]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost14checked_deleteIN5mongo18DBClientConnectionEEEvPT_+0x2a) [0x8302b0]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost6detail17sp_counted_impl_pIN5mongo18DBClientConnectionEE7disposeEv+0x1c) [0x83299c]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost6detail15sp_counted_base7releaseEv+0x42) [0x760fe6]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost6detail12shared_countD1Ev+0x27) [0x761075]
 m30999|  /home/aaron/mongo3/mongos(_ZN5boost10shared_ptrIN5mongo18DBClientConnectionEED1Ev+0x1c) [0x82cf68]
 m30999|  /home/aaron/mongo3/mongos(_ZN5mongo17ReplicaSetMonitor4NodeD1Ev+0x2d) [0x82dd3d]
 m30999|  /home/aaron/mongo3/mongos(_ZSt8_DestroyIN5mongo17ReplicaSetMonitor4NodeEEvPT_+0x18) [0x83215c]
 m30999|  /home/aaron/mongo3/mongos(_ZNSt12_Destroy_auxILb0EE9__destroyIPN5mongo17ReplicaSetMonitor4NodeEEEvT_S6_+0x1e) [0x831768]
 m30999|  /home/aaron/mongo3/mongos(_ZSt8_DestroyIPN5mongo17ReplicaSetMonitor4NodeEEvT_S4_+0x23) [0x830537]
 m30999|  /home/aaron/mongo3/mongos(_ZSt8_DestroyIPN5mongo17ReplicaSetMonitor4NodeES2_EvT_S4_RSaIT0_E+0x27) [0x82f2b7]
 
 
 
 
 
 m31200| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m31200| 0x9256bf 0x934b39 0x8a719f 0x9485b8 0x946976 0x9469fe 0x965178 0x963881 0x961bad 0x961c6c 0x973d90 0x97647c 0x8a4ac6 0x8a4b55 0x970a48 0x97181d 0x975c3c 0x975248 0x974017 0x972d97 
 m31200| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m31200| 0x9256bf 0x934b39 0x8a719f 0x91ce33 0x924b81 0x924a8b 0x924a0c 0x924780 0x7f25df1c3a55 0x33c5206ccb 0x33c4ae0c2d 
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x9256bf]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev+0x51) [0x934b39]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lockC1ERS0_+0x29) [0x8a719f]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x29f) [0x91ce33]
 m31200|  /home/aaron/mongo3/mongod(_ZNK5boost4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS3_9JobStatusEEEEclEPS3_S6_+0x7b) [0x924b81]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost3_bi5list2INS0_5valueIPN5mongo13BackgroundJobEEENS2_INS_10shared_ptrINS4_9JobStatusEEEEEEclINS_4_mfi3mf1IvS4_S9_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x77) [0x924a8b]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS5_9JobStatusEEEEENS0_5list2INS0_5valueIPS5_EENSB_IS8_EEEEEclEv+0x3c) [0x924a0c]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x1e) [0x924780]
 m31200|  /usr/lib64/libboost_thread-mt.so.1.44.0(thread_proxy+0x65) [0x7f25df1c3a55]
 m31200|  /lib64/libpthread.so.0() [0x33c5206ccb]
 m31200|  /lib64/libc.so.6(clone+0x6d) [0x33c4ae0c2d]
 m31200| Logstream::get called in uninitialized state
 
 
 
 
 
 
 m31200| _DEBUG warning trying to lock a mongo::mutex during static shutdown
 m31200| 0x9256bf 0x934b39 0x8a719f 0x9485b8 0x946976 0x9469fe 0x965178 0x963881 0x961bad 0x961c6c 0x973d90 0x97647c 0x8a4ac6 0x8a4b55 0x970a48 0x97181d 0x975c3c 0x975248 0x974017 0x972d97 
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x9256bf]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev+0x51) [0x934b39]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lockC1ERS0_+0x29) [0x8a719f]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo9Logstream5flushEPNS_3TeeE+0x2b8) [0x8a83be]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo9LogstreamlsEPFRSoS1_E+0x43) [0x8a7dc5]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo6ClientD1Ev+0xe5) [0xc19be7]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo6ClientD0Ev+0x18) [0xc19d98]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost19thread_specific_ptrIN5mongo6ClientEE11delete_dataclEPv+0x30) [0xc20cbc]
 m31200|  /usr/lib64/libboost_thread-mt.so.1.44.0(tls_destructor+0xd3) [0x7f25df1c0a23]
 m31200|  /usr/lib64/libboost_thread-mt.so.1.44.0(thread_proxy+0x6f) [0x7f25df1c3a5f]
 m31200|  /lib64/libpthread.so.0() [0x33c5206ccb]
 m31200|  /lib64/libc.so.6(clone+0x6d) [0x33c4ae0c2d]
 
 
 
 
 
 m31200| Thu Dec 29 00:58:14 ERROR: Client::shutdown not called: slaveTracking
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x9256bf]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lock15PostStaticCheckC1Ev+0x51) [0x934b39]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5mutex11scoped_lockC1ERS0_+0x29) [0x8a719f]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo5Ports5eraseEPNS_13MessagingPortE+0x28) [0x9485b8]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo13MessagingPortD1Ev+0x74) [0x946976]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo13MessagingPortD0Ev+0x18) [0x9469fe]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost14checked_deleteIN5mongo13MessagingPortEEEvPT_+0x2a) [0x965178]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost10scoped_ptrIN5mongo13MessagingPortEED1Ev+0x1b) [0x963881]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo18DBClientConnectionD1Ev+0x9d) [0x961bad]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo18DBClientConnectionD0Ev+0x18) [0x961c6c]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost14checked_deleteIN5mongo18DBClientConnectionEEEvPT_+0x2a) [0x973d90]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost6detail17sp_counted_impl_pIN5mongo18DBClientConnectionEE7disposeEv+0x1c) [0x97647c]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost6detail15sp_counted_base7releaseEv+0x42) [0x8a4ac6]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost6detail12shared_countD1Ev+0x27) [0x8a4b55]
 m31200|  /home/aaron/mongo3/mongod(_ZN5boost10shared_ptrIN5mongo18DBClientConnectionEED1Ev+0x1c) [0x970a48]
 m31200|  /home/aaron/mongo3/mongod(_ZN5mongo17ReplicaSetMonitor4NodeD1Ev+0x2d) [0x97181d]
 m31200|  /home/aaron/mongo3/mongod(_ZSt8_DestroyIN5mongo17ReplicaSetMonitor4NodeEEvPT_+0x18) [0x975c3c]
 m31200|  /home/aaron/mongo3/mongod(_ZNSt12_Destroy_auxILb0EE9__destroyIPN5mongo17ReplicaSetMonitor4NodeEEEvT_S6_+0x1e) [0x975248]
 m31200|  /home/aaron/mongo3/mongod(_ZSt8_DestroyIPN5mongo17ReplicaSetMonitor4NodeEEvT_S4_+0x23) [0x974017]
 m31200|  /home/aaron/mongo3/mongod(_ZSt8_DestroyIPN5mongo17ReplicaSetMonitor4NodeES2_EvT_S4_RSaIT0_E+0x27) [0x972d97]
 
 
 
 
 
*** glibc detected *** /home/aaron/mongo3/mongos: free(): invalid pointer: 0x0000000000e50840 ***
======= Backtrace: =========
/lib64/libc.so.6[0x33c4a76d63]
/home/aaron/mongo3/mongos(_ZN5mongo17CursorTimeoutTaskD0Ev+0x24)[0xa6add0]
/home/aaron/mongo3/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x31c)[0x7d93d0]
/home/aaron/mongo3/mongos(_ZNK5boost4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS3_9JobStatusEEEEclEPS3_S6_+0x7b)[0x7e10a1]
/home/aaron/mongo3/mongos(_ZN5boost3_bi5list2INS0_5valueIPN5mongo13BackgroundJobEEENS2_INS_10shared_ptrINS4_9JobStatusEEEEEEclINS_4_mfi3mf1IvS4_S9_EENS0_5list0EEEvNS0_4typeIvEERT_RT0_i+0x77)[0x7e0fab]
/home/aaron/mongo3/mongos(_ZN5boost3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS5_9JobStatusEEEEENS0_5list2INS0_5valueIPS5_EENSB_IS8_EEEEEclEv+0x3c)[0x7e0f2c]
/home/aaron/mongo3/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x1e)[0x7e0ca0]
/usr/lib64/libboost_thread-mt.so.1.44.0(thread_proxy+0x65)[0x7f7dac6f1a55]
/lib64/libpthread.so.0[0x33c5206ccb]
/lib64/libc.so.6(clone+0x6d)[0x33c4ae0c2d]
======= Memory map: ========
00400000-00c33000 r-xp 00000000 ca:01 11452494                           /home/aaron/mongo3/mongos
00e32000-00e4e000 rw-p 00832000 ca:01 11452494                           /home/aaron/mongo3/mongos
00e4e000-00e51000 rw-p 00000000 00:00 0
020ee000-02130000 rw-p 00000000 00:00 0                                  [heap]
33c4600000-33c461f000 r-xp 00000000 ca:01 11846588                       /lib64/ld-2.13.so
33c481e000-33c481f000 r--p 0001e000 ca:01 11846588                       /lib64/ld-2.13.so
33c481f000-33c4820000 rw-p 0001f000 ca:01 11846588                       /lib64/ld-2.13.so
33c4820000-33c4821000 rw-p 00000000 00:00 0
33c4a00000-33c4b91000 r-xp 00000000 ca:01 11846732                       /lib64/libc-2.13.so
33c4b91000-33c4d91000 ---p 00191000 ca:01 11846732                       /lib64/libc-2.13.so
33c4d91000-33c4d95000 r--p 00191000 ca:01 11846732                       /lib64/libc-2.13.so
33c4d95000-33c4d96000 rw-p 00195000 ca:01 11846732                       /lib64/libc-2.13.so
33c4d96000-33c4d9c000 rw-p 00000000 00:00 0
33c5200000-33c5217000 r-xp 00000000 ca:01 11846584                       /lib64/libpthread-2.13.so
33c5217000-33c5416000 ---p 00017000 ca:01 11846584                       /lib64/libpthread-2.13.so
33c5416000-33c5417000 r--p 00016000 ca:01 11846584                       /lib64/libpthread-2.13.so
33c5417000-33c5418000 rw-p 00017000 ca:01 11846584                       /lib64/libpthread-2.13.so
33c5418000-33c541c000 rw-p 00000000 00:00 0
33c5e00000-33c5e07000 r-xp 00000000 ca:01 11846767                       /lib64/librt-2.13.so
33c5e07000-33c6006000 ---p 00007000 ca:01 11846767                       /lib64/librt-2.13.so
33c6006000-33c6007000 r--p 00006000 ca:01 11846767                       /lib64/librt-2.13.so
33c6007000-33c6008000 rw-p 00007000 ca:01 11846767                       /lib64/librt-2.13.so
33c6200000-33c6284000 r-xp 00000000 ca:01 11846765                       /lib64/libm-2.13.so
33c6284000-33c6483000 ---p 00084000 ca:01 11846765                       /lib64/libm-2.13.so
33c6483000-33c6484000 r--p 00083000 ca:01 11846765                       /lib64/libm-2.13.so
33c6484000-33c6485000 rw-p 00084000 ca:01 11846765                       /lib64/libm-2.13.so
33c6600000-33c6615000 r-xp 00000000 ca:01 11846727                       /lib64/libgcc_s-4.5.1-20100924.so.1
33c6615000-33c6814000 ---p 00015000 ca:01 11846727                       /lib64/libgcc_s-4.5.1-20100924.so.1
33c6814000-33c6815000 rw-p 00014000 ca:01 11846727                       /lib64/libgcc_s-4.5.1-20100924.so.1
33c7e00000-33c7ee8000 r-xp 00000000 ca:01 11994216                       /usr/lib64/libstdc++.so.6.0.14
33c7ee8000-33c80e8000 ---p 000e8000 ca:01 11994216                       /usr/lib64/libstdc++.so.6.0.14
33c80e8000-33c80f0000 r--p 000e8000 ca:01 11994216                       /usr/lib64/libstdc++.so.6.0.14
33c80f0000-33c80f2000 rw-p 000f0000 ca:01 11994216                       /usr/lib64/libstdc++.so.6.0.14
33c80f2000-33c8107000 rw-p 00000000 00:00 0
7f7d78000000-7f7d78021000 rw-p 00000000 00:00 0
7f7d78021000-7f7d7c000000 ---p 00000000 00:00 0
7f7d7c000000-7f7d7c021000 rw-p 00000000 00:00 0
7f7d7c021000-7f7d80000000 ---p 00000000 00:00 0
7f7d80000000-7f7d80021000 rw-p 00000000 00:00 0
7f7d80021000-7f7d84000000 ---p 00000000 00:00 0
7f7d84000000-7f7d84021000 rw-p 00000000 00:00 0
7f7d84021000-7f7d88000000 ---p 00000000 00:00 0
7f7d88000000-7f7d88021000 rw-p 00000000 00:00 0
7f7d88021000-7f7d8c000000 ---p 00000000 00:00 0
7f7d8c000000-7f7d8c021000 rw-p 00000000 00:00 0
7f7d8c021000-7f7d90000000 ---p 00000000 00:00 0
7f7d90000000-7f7d90021000 rw-p 00000000 00:00 0
7f7d90021000-7f7d94000000 ---p 00000000 00:00 0
7f7d94ffa000-7f7d94ffb000 ---p 00000000 00:00 0
7f7d94ffb000-7f7d957fb000 rw-p 00000000 00:00 0
7f7d957fb000-7f7d957fc000 ---p 00000000 00:00 0
7f7d957fc000-7f7d95ffc000 rw-p 00000000 00:00 0
7f7d95ffc000-7f7d95ffd000 ---p 00000000 00:00 0
7f7d95ffd000-7f7d967fd000 rw-p 00000000 00:00 0
7f7d967fd000-7f7d967fe000 ---p 00000000 00:00 0
7f7d967fe000-7f7d96ffe000 rw-p 00000000 00:00 0
7f7d96ffe000-7f7d96fff000 ---p 00000000 00:00 0
7f7d96fff000-7f7d977ff000 rw-p 00000000 00:00 0
7f7d977ff000-7f7d97800000 ---p 00000000 00:00 0
7f7d97800000-7f7d98000000 rw-p 00000000 00:00 0
7f7d98000000-7f7d98021000 rw-p 00000000 00:00 0
7f7d98021000-7f7d9c000000 ---p 00000000 00:00 0
7f7d9c000000-7f7d9c021000 rw-p 00000000 00:00 0
7f7d9c021000-7f7da0000000 ---p 00000000 00:00 0
7f7da0000000-7f7da0021000 rw-p 00000000 00:00 0
7f7da0021000-7f7da4000000 ---p 00000000 00:00 0
7f7da411a000-7f7da411b000 ---p 00000000 00:00 0
7f7da411b000-7f7da419b000 rw-p 00000000 00:00 0
7f7da419b000-7f7da419c000 ---p 00000000 00:00 0
7f7da419c000-7f7da499c000 rw-p 00000000 00:00 0
7f7da499c000-7f7da499d000 ---p 00000000 00:00 0
7f7da499d000-7f7da519d000 rw-p 00000000 00:00 0
7f7da519d000-7f7da519e000 ---p 00000000 00:00 0
7f7da519e000-7f7da599e000 rw-p 00000000 00:00 0
7f7da599e000-7f7da599f000 ---p 00000000 00:00 0
7f7da599f000-7f7da619f000 rw-p 00000000 00:00 0
7f7da619f000-7f7da61aa000 r-xp 00000000 ca:01 11846724                   /lib64/libnss_files-2.13.so
7f7da61aa000-7f7da63a9000 ---p 0000b000 ca:01 11846724                   /lib64/libnss_files-2.13.so
7f7da63a9000-7f7da63aa000 r--p 0000a000 ca:01 11846724                   /lib64/libnss_files-2.13.so
7f7da63aa000-7f7da63ab000 rw-p 0000b000 ca:01 11846724                   /lib64/libnss_files-2.13.so
7f7da63b5000-7f7dac246000 r--p 00000000 ca:01 12044451                   /usr/lib/locale/locale-archive
7f7dac246000-7f7dac24e000 rw-p 00000000 00:00 0

Generated at Thu Feb 08 03:06:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.