-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Networking, Sharding
-
None
-
Fully Compatible
-
ALL
-
Platform A (10/09/15)
-
0
As evidenced by recent failures in the sharding jscore passthrough suites, shutting down the config servers can cause mongos can call terminate() when an exception of type std::bad_function_call is active.
Example failure of sharded_collections_jscore_passthrough on Linux:
Example failure of sharding_jscore_passthrough on SSL RHEL 5.5:
The sequence of relevant events seems to be as follows:
- resmoke.py sends a SIGTERM to all three config servers
- one of the mongos client threads prints the warning "failed to close stream:Transport endpoint is not connected"
- shortly after, the same client thread calls terminate()
- resmoke.py attempts to send a SIGTERM to mongos, but discovers that it has crashed
Excerpt from first failure above:
[ShardedClusterFixture:job0:mongos] 2015-09-29T19:23:07.571+0000 F - [thread2] terminate() called. An exception is active; attempting to gather more information
[ShardedClusterFixture:job0:mongos] 2015-09-29T19:23:07.571+0000 F - [thread2] std::exception::what(): bad_function_call
[ShardedClusterFixture:job0:mongos] Actual exception type: std::bad_function_call
[ShardedClusterFixture:job0:mongos]
[ShardedClusterFixture:job0:mongos] 0xba63e2 0xba5d12 0x2b9e0fc9ae46 0x2b9e0fc9ae73 0xd0a795 0x2b9e1037683d 0x2b9e10661fdd
[ShardedClusterFixture:job0:mongos] ----- BEGIN BACKTRACE -----
[ShardedClusterFixture:job0:mongos] {"backtrace":[{"b":"400000","o":"7A63E2"},{"b":"400000","o":"7A5D12"},{"b":"2B9E0FBDE000","o":"BCE46"},{"b":"2B9E0FBDE000","o":"BCE73"},{"b":"400000","o":"90A795"},{"b":"2B9E10370000","o":"683D"},{"b":"2B9E1058D000","o":"D4FDD"}],"processInfo":{ "mongodbVersion" : "3.1.9-pre-", "gitVersion" : "05e9080be96f05fcf4ed74b996308edd14f2cec1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.18-194.el5xen", "version" : "#1 SMP Tue Mar 16 22:01:26 EDT 2010", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "2B9E0F7D1000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "2B9E0F9DA000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "2B9E0FBDE000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "2B9E0FEDF000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "2B9E10162000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "2B9E10370000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "2B9E1058D000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "2B9E0F5B3000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
[ShardedClusterFixture:job0:mongos] mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xba63e2]
[ShardedClusterFixture:job0:mongos] mongos(+0x7A5D12) [0xba5d12]
[ShardedClusterFixture:job0:mongos] libstdc++.so.6(+0xBCE46) [0x2b9e0fc9ae46]
[ShardedClusterFixture:job0:mongos] libstdc++.so.6(+0xBCE73) [0x2b9e0fc9ae73]
[ShardedClusterFixture:job0:mongos] mongos(+0x90A795) [0xd0a795]
[ShardedClusterFixture:job0:mongos] libpthread.so.0(+0x683D) [0x2b9e1037683d]
[ShardedClusterFixture:job0:mongos] libc.so.6(clone+0x6D) [0x2b9e10661fdd]
[ShardedClusterFixture:job0:mongos] ----- END BACKTRACE -----
Assigning to adam.midvidy for triage.