[SERVER-34530] Shard server crashes after access violation on Windows v3.7.4-6-g228106a741 Created: 17/Apr/18  Updated: 11/Jun/19  Resolved: 18/Apr/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Shane Harvey Assignee: Esha Maharishi (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongo-orchestration.log     Text File mongod-1026.log     Text File mongod-27217.log     Text File mongod-27218.log     Text File mongos-27017.log    
Issue Links:
Duplicate
duplicates SERVER-32677 Segmentation fault converting Replica... Closed
Related
is related to PYTHON-1531 Mongo-orchestration fails on Windows ... Closed
Operating System: ALL
Sprint: Sharding 2018-04-23
Participants:

 Description   

In the following Python driver test run a shard server crashed after unhandled exception: https://evergreen.mongodb.com/task/mongo_python_driver_tests_windows_vs2015_python_version__auth_ssl~auth_ssl_windows_vs2015_python_version~2.6_test_latest_sharded_cluster_d1ee646d7203bffba4c65edf66d3db1cb33518b3_18_04_17_20_23_52

2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] MongoDB starting : pid=4772 port=27217 dbpath=c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\mongo-jufiiu 64-bit host=WIN-T1NSTPV2UDM
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] db version v3.7.4-6-g228106a741
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] git version: 228106a7416288ad6263ab67878477cc23e9c23f
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] modules: enterprise 
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten] build environment:
2018-04-17T20:33:43.398+0000 I CONTROL  [initandlisten]     distmod: windows-64
2018-04-17T20:33:43.399+0000 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-17T20:33:43.399+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-17T20:33:43.399+0000 I CONTROL  [initandlisten] options: { config: "c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\mongo-y8jggw", net: { bindIp: "127.0.0.1,::1", compression: { compressors: "zlib,snappy,noop" }, ipv6: true, port: 27217, ssl: { CAFile: "C:/data/mci/71adf00840eee64d74f9bd1375d5226e/drivers-tools/.evergreen/x509gen/ca.pem", PEMKeyFile: "C:/data/mci/71adf00840eee64d74f9bd1375d5226e/drivers-tools/.evergreen/x509gen/server.pem", mode: "requireSSL", weakCertificateValidation: true } }, replication: { enableMajorityReadConcern: true, oplogSizeMB: 100, replSet: "d32c6dd4-eaa7-4548-82b7-edfa1b821bcc" }, security: { keyFile: "c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\tmphikgfm\key" }, setParameter: { enableTestCommands: "1" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\mongo-jufiiu" }, systemLog: { destination: "file", path: "c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\mongo-jufiiu\mongod.log" } }
2018-04-17T20:33:43.400+0000 I STORAGE  [initandlisten] Detected data files in c:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\.evergreen\orchestration\db\mongo-jufiiu created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-17T20:33:43.400+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3327M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-04-17T20:33:43.431+0000 I STORAGE  [initandlisten] WiredTiger message [1523997223:431856][4772:1995256704], txn-recover: Set global recovery timestamp: 0
2018-04-17T20:33:43.460+0000 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2018-04-17T20:33:43.460+0000 I STORAGE  [initandlisten] The size storer reports that the oplog contains 0 records totaling to 0 bytes
2018-04-17T20:33:43.460+0000 I STORAGE  [initandlisten] Scanning the oplog to determine where to place markers for truncation
2018-04-17T20:33:43.479+0000 I CONTROL  [initandlisten] 
.
.
.
2018-04-17T20:38:44.830+0000 I CONTROL  [thread582] *** unhandled exception (access violation) at 0x000007FEEF848BD6, terminating
2018-04-17T20:38:44.830+0000 I CONTROL  [thread582] *** access violation was a read from 0x40
2018-04-17T20:38:44.830+0000 I CONTROL  [thread582] *** stack trace for unhandled exception:
2018-04-17T20:38:45.004+0000 I -        [thread582] MSVCP140.dll                                                                                              Mtx_destroy+0x70x
mongod.exe    ...\src\mongo\s\catalog_cache.cpp(363)                                                      mongo::CatalogCache::invalidateShardedCollection+0x63x
mongod.exe    ...\src\mongo\s\catalog_cache.cpp(270)                                                      mongo::CatalogCache::getShardedCollectionRoutingInfoWithRefresh+0x77x
mongod.exe    ...\src\mongo\db\sessions_collection_sharded.cpp(60)                                        mongo::SessionsCollectionSharded::_checkCacheForSessionsCollection+0x166x
mongod.exe    ...\src\mongo\db\sessions_collection_sharded.cpp(75)                                        mongo::SessionsCollectionSharded::setupSessionsCollection+0x22x
mongod.exe    ...\src\mongo\db\logical_session_cache_impl.cpp(264)                                        mongo::LogicalSessionCacheImpl::_refresh+0x341x
mongod.exe    ...\src\mongo\db\logical_session_cache_impl.cpp(162)                                        mongo::LogicalSessionCacheImpl::_periodicRefresh+0x39x
mongod.exe    ...\src\mongo\util\periodic_runner_asio.cpp(96)                                             <lambda_aca5a7c3ea2730571ac362eb390ff069>::operator()+0x442x
mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional(214)              std::_Func_impl<<lambda_aca5a7c3ea2730571ac362eb390ff069>,std::allocator<int>,void,std::error_code>::_Do_call+0x26x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\handler_invoke_hook.hpp(69)               asio::asio_handler_invoke<asio::detail::binder1<std::function<void __cdecl(std::error_code)>,std::error_code> >+0x55x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\impl\strand_service.hpp(61)        asio::detail::strand_service::dispatch<asio::detail::binder1<std::function<void __cdecl(std::error_code)>,std::error_code> >+0x98x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\wrapped_handler.hpp(86)            asio::detail::wrapped_handler<asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running>::operator()<std::error_code>+0x72x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\completion_handler.hpp(69)         asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running>,std::error_code>,std::function<void __cdecl(std::error_code)> > >::do_complete+0x248x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\impl\strand_service.hpp(87)        asio::detail::strand_service::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running>,std::error_code>,std::function<void __cdecl(std::error_code)> > >+0x294x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\wrapped_handler.hpp(231)           asio::detail::asio_handler_invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running>,std::error_code>,asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running>+0x147x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\wait_handler.hpp(71)               asio::detail::wait_handler<asio::detail::wrapped_handler<asio::io_context::strand,std::function<void __cdecl(std::error_code)>,asio::detail::is_continuation_if_running> >::do_complete+0x203x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\impl\win_iocp_io_context.ipp(420)  asio::detail::win_iocp_io_context::do_one+0x711x
mongod.exe    ...\src\third_party\asio-master\asio\include\asio\detail\impl\win_iocp_io_context.ipp(162)  asio::detail::win_iocp_io_context::run+0x205x
mongod.exe    ...\src\mongo\util\periodic_runner_asio.cpp(114)                                            <lambda_dbfc2572039c3735bfc79c6454ca3cfd>::operator()+0x227x
mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)             std::_LaunchPad<std::unique_ptr<std::tuple<<lambda_dbfc2572039c3735bfc79c6454ca3cfd> >,std::default_delete<std::tuple<<lambda_dbfc2572039c3735bfc79c6454ca3cfd> > > > >::_Run+0x119x
mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)             std::_Pad::_Call_func+0x9x
ucrtbase.DLL                                                                                              crt_at_quick_exit+0x125x
kernel32.dll                                                                                              BaseThreadInitThunk+0x13x
2018-04-17T20:38:45.004+0000 I CONTROL  [thread582] writing minidump diagnostic file C:\data\mci\71adf00840eee64d74f9bd1375d5226e\drivers-tools\mongodb\bin\mongod.2018-04-17T20-38-45.mdmp
2018-04-17T20:38:45.985+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:50665 #582 (9 connections now open)
2018-04-17T20:38:45.985+0000 I CONTROL  [thread582] *** immediate exit due to unhandled exception



 Comments   
Comment by Shane Harvey [ 19/Apr/18 ]

I'm interested in what the steps taken by mongo-orchestration and the python test are, since it takes a bit of manipulation to trigger the invalid access in SERVER-32677. It'd be useful to know whether it's the same manipulation, or if this can be triggered another way.

Mongo-orchestration did add this node as a shard. The real issue here is SERVER-34575.

Comment by Esha Maharishi (Inactive) [ 18/Apr/18 ]

I am closing this as a dupe of SERVER-32677, which kaloian.manassiev and blake.oler are working on, but am still interested in how this got triggered by the drivers test

Comment by Esha Maharishi (Inactive) [ 18/Apr/18 ]

Hmm. Actually, I can repro this locally by simply starting up a mongod as a --shardsvr (no need to start any other nodes).

It probably doesn't show up in any of the server team's test infrastructure, because it takes 5 minutes for this periodic refresh to run, and we probably don't have any tests that start a shardsvr without adding it as a shard for 5 minutes.

$ ./mongod --replSet test --shardsvr
2018-04-18T10:35:10.827-0400 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] MongoDB starting : pid=31456 port=27018 dbpath=/data/db 64-bit host=eshamaharishi-X10DAi
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] DEBUG build (which is slower)
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] db version v0.0.0
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] git version: unknown
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] allocator: system
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] modules: enterprise ninja
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] build environment:
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten]     distarch: x86_64
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten]     target_arch: x86_64
2018-04-18T10:35:10.856-0400 I CONTROL  [initandlisten] options: { replication: { replSet: "test" }, sharding: { clusterRole: "shardsvr" } }
2018-04-18T10:35:10.856-0400 I STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-04-18T10:35:10.857-0400 I STORAGE  [initandlisten]
2018-04-18T10:35:10.857-0400 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-04-18T10:35:10.857-0400 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-04-18T10:35:10.857-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31639M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-04-18T10:35:12.076-0400 I STORAGE  [initandlisten] WiredTiger message [1524062112:76656][31456:0x7f4a0ef359c0], txn-recover: Main recovery loop: starting at 15/10752
2018-04-18T10:35:12.179-0400 I STORAGE  [initandlisten] WiredTiger message [1524062112:178983][31456:0x7f4a0ef359c0], txn-recover: Recovering log 15 through 16
2018-04-18T10:35:12.250-0400 I STORAGE  [initandlisten] WiredTiger message [1524062112:250057][31456:0x7f4a0ef359c0], txn-recover: Recovering log 16 through 16
2018-04-18T10:35:12.301-0400 I STORAGE  [initandlisten] WiredTiger message [1524062112:301679][31456:0x7f4a0ef359c0], txn-recover: Set global recovery timestamp: 0
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] ** WARNING: This server is bound to localhost.
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          Remote systems will be unable to connect to this server.
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          Start the server with --bind_ip <address> to specify which IP
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          addresses it should serve responses from, or with --bind_ip_all to
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          bind to all interfaces. If this behavior is desired, start the
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten] **          server with --bind_ip 127.0.0.1 to disable this warning.
2018-04-18T10:35:12.515-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-18T10:35:12.517-0400 I CONTROL  [initandlisten]
2018-04-18T10:35:12.640-0400 W SHARDING [initandlisten] Started with --shardsvr, but no shardIdentity document was found on disk in admin.system.version. This most likely means this server has not yet been added to a sharded cluster.
2018-04-18T10:35:12.642-0400 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2018-04-18T10:35:12.644-0400 I STORAGE  [initandlisten] createCollection: local.replset.checkpointTimestamp with generated UUID: c967dea0-44a0-4ca8-ab28-2f9251f1f9ec
2018-04-18T10:35:12.729-0400 I REPL     [initandlisten] Did not find local voted for document at startup.
2018-04-18T10:35:12.731-0400 I REPL     [initandlisten] Rollback ID is 0
2018-04-18T10:35:12.731-0400 I REPL     [initandlisten] Did not find local replica set configuration document at startup;  NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2018-04-18T10:35:12.733-0400 I NETWORK  [initandlisten] waiting for connections on port 27018
2018-04-18T10:37:12.733-0400 F -        [thread1] Invalid access at address: 0x50
2018-04-18T10:37:12.917-0400 F -        [thread1] Got signal: 11 (Segmentation fault).
 0x563ae4d993e3 0x563ae4d98eb5 0x563ae4d988c9 0x7f4a0bc44390 0x7f4a0bc3cd44 0x563ae25fe393 0x563ae25ffb05 0x563ae265cd93 0x563ae41538f4 0x563ae4153cf7 0x563ae3ebcfb1 0x563ae3ebd224 0x563ae3f3968c 0x563ae3f3abad 0x563ae3f3bf10 0x563ae3f3bdc2 0x563ae3e72414 0x563ae3e6b3d0 0x563ae3e6afca 0x563ae4450c77 0x563ae4450bed 0x563ae4450bb1 0x563ae4450b92 0x563ae4450b4d 0x563ae4450830 0x563ae44505c7 0x563ae44504a5 0x563ae445040f 0x563ae445039f 0x563ae4450379 0x563ae4450351 0x563ae4450332 0x563ae44502ed 0x563ae444ffb0 0x563ae4451641 0x563ae44501d3 0x563ae444fe8a 0x563ae444fbd5 0x563ae444fb39 0x563ae444fac0 0x563ae444f901 0x563ae444f675 0x563ae4c7e7b7 0x563ae4c6fed8 0x563ae4c65fd6 0x563ae4c66131 0x563ae3e6d356 0x563ae3e6d1c5 0x563ae3e6d195 0x563ae3e6d089 0x563ae4f61cf0 0x7f4a0bc3a6ba 0x7f4a0b97082d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"563AE064C000","o":"474D3E3","s":"_ZN5mongo15printStackTraceERSo"},{"b":"563AE064C000","o":"474CEB5"},{"b":"563AE064C000","o":"474C8C9"},{"b":"7F4A0BC33000","o":"11390"},{"b":"7F4A0BC33000","o":"9D44","s":"pthread_mutex_lock"},{"b":"563AE064C000","o":"1FB2393"},{"b":"563AE064C000","o":"1FB3B05","s":"_ZNSt5mutex4lockEv"},{"b":"563AE064C000","o":"2010D93","s":"_ZNSt10lock_guardISt5mutexEC2ERS0_"},{"b":"563AE064C000","o":"3B078F4","s":"_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE"},{"b":"563AE064C000","o":"3B07CF7","s":"_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE"},{"b":"563AE064C000","o":"3870FB1","s":"_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE"},{"b":"563AE064C000","o":"3871224","s":"_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE"},{"b":"563AE064C000","o":"38ED68C","s":"_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE"},{"b":"563AE064C000","o":"38EEBAD","s":"_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE"},{"b":"563AE064C000","o":"38EFF10"},{"b":"563AE064C000","o":"38EFDC2"},{"b":"563AE064C000","o":"3826414","s":"_ZNKSt8functionIFvPN5mongo6ClientEEEclES2_"},{"b":"563AE064C000","o":"381F3D0"},{"b":"563AE064C000","o":"381EFCA"},{"b":"563AE064C000","o":"3E04C77","s":"_ZNKSt8functionIFvSt10error_codeEEclES0_"},{"b":"563AE064C000","o":"3E04BED","s":"_ZN4asio6detail7binder1ISt8functionIFvSt10error_codeEES3_EclEv"},{"b":"563AE064C000","o":"3E04BB1","s":"_ZN4asio19asio_handler_invokeINS_6detail7binder1ISt8functionIFvSt10error_codeEES4_EEEEvRT_z"},{"b":"563AE064C000","o":"3E04B92","s":"_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1ISt8functionIFvSt10error_codeEES5_EES7_EEvRT_RT0_"},{"b":"563AE064C000","o":"3E04B4D","s":"_ZN4asio6detail19asio_handler_invokeINS0_7binder1ISt8functionIFvSt10error_codeEES4_EES6_S4_EEvRT_PNS2_IT0_T1_EE"},{"b":"563AE064C000","o":"3E04830","s":"_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1ISt8functionIFvSt10error_codeEES5_EES8_EEvRT_RT0_"},{"b":"563AE064C000","o":"3E045C7","s":"_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_"},{"b":"563AE064C000","o":"3E044A5","s":"_ZN4asio10io_context6strand8dispatchINS_6detail7binder1ISt8functionIFvSt10error_codeEES6_EEEENS_12async_resultINSt5decayIT_E4typeEFvvEE11return_typeEOSC_"},{"b":"563AE064C000","o":"3E0440F","s":"_ZN4asio6detail15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEclIS5_EEvRKT_"},{"b":"563AE064C000","o":"3E0439F","s":"_ZN4asio6detail7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES6_EclEv"},{"b":"563AE064C000","o":"3E04379","s":"_ZN4asio6detail17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES7_EES9_EclEv"},{"b":"563AE064C000","o":"3E04351","s":"_ZN4asio19asio_handler_invokeINS_6detail17rewrapped_handlerINS1_7binder1INS1_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS1_26is_continuation_if_runningEEES8_EESA_EEEEvRT_z"},{"b":"563AE064C000","o":"3E04332","s":"_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail17rewrapped_handlerINS2_7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES9_EESB_EESB_EEvRT_RT0_"},{"b":"563AE064C000","o":"3E042ED","s":"_ZN4asio6detail19asio_handler_invokeINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EESD_SA_EEvRT_PNS2_IT0_T1_EE"},{"b":"563AE064C000","o":"3E03FB0","s":"_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail17rewrapped_handlerINS2_7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES9_EESB_EESF_EEvRT_RT0_"},{"b":"563AE064C000","o":"3E05641","s":"_ZN4asio6detail12handler_workINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EENS_15system_executorEE8completeISE_EEvRT_RSE_"},{"b":"563AE064C000","o":"3E041D3","s":"_ZN4asio6detail18completion_handlerINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EEE11do_completeEPvPNS0_19scheduler_operationERKS8_m"},{"b":"563AE064C000","o":"3E03E8A","s":"_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_"},{"b":"563AE064C000","o":"3E03BD5","s":"_ZN4asio10io_context6strand8dispatchINS_6detail17rewrapped_handlerINS3_7binder1INS3_15wrapped_handlerIS1_St8functionIFvSt10error_codeEENS3_26is_continuation_if_runningEEES8_EESA_EEEENS_12async_resultINSt5decayIT_E4typeEFvvEE11return_typeEOSH_"},{"b":"563AE064C000","o":"3E03B39","s":"_ZN4asio6detail19asio_handler_invokeINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES7_EES5_S9_SA_EEvRT_PNS3_IT0_T1_T2_EE"},{"b":"563AE064C000","o":"3E03AC0","s":"_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES8_EESC_EEvRT_RT0_"},{"b":"563AE064C000","o":"3E03901","s":"_ZN4asio6detail12handler_workINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEENS_15system_executorEE8completeINS0_7binder1ISA_S6_EEEEvRT_RSA_"},{"b":"563AE064C000","o":"3E03675","s":"_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m"},{"b":"563AE064C000","o":"46327B7","s":"_ZN4asio6detail19scheduler_operation8completeEPvRKSt10error_codem"},{"b":"563AE064C000","o":"4623ED8","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"563AE064C000","o":"4619FD6","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"563AE064C000","o":"461A131","s":"_ZN4asio10io_context3runERSt10error_code"},{"b":"563AE064C000","o":"3821356"},{"b":"563AE064C000","o":"38211C5"},{"b":"563AE064C000","o":"3821195"},{"b":"563AE064C000","o":"3821089"},{"b":"563AE064C000","o":"4915CF0"},{"b":"7F4A0BC33000","o":"76BA"},{"b":"7F4A0B86A000","o":"10682D","s":"clone"}],"processInfo":{ "mongodbVersion" : "0.0.0", "gitVersion" : "unknown", "compiledModules" : [ "enterprise", "ninja" ], "uname" : { "sysname" : "Linux", "release" : "4.4.0-71-generic", "version" : "#92-Ubuntu SMP Fri Mar 24 12:59:01 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "563AE064C000", "elfType" : 3, "buildId" : "7B0A1DF3C3F274BE410D1D71BA351293544096EE" }, { "b" : "7FFE14455000", "elfType" : 3, "buildId" : "C0FFFA04E21227146887D2DD25FBB054425BF143" }, { "b" : "7F4A0E8EB000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "8FE272AED3883FEB05AC784E50D9D8F0FAC4E899" }, { "b" : "7F4A0E6DC000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "1A855916EA4E082827D0599C11C0E82D98284118" }, { "b" : "7F4A0E4CF000", "path" : "/lib/x86_64-linux-gnu/libpci.so.3", "elfType" : 3, "buildId" : "3F90C8FDF5FC50256DD803E6F5EBED276A8193A6" }, { "b" : "7F4A0E2CB000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C0C5B7F18348654040534B050B110D32A19EA38D" }, { "b" : "7F4A0E062000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "901744249A5926FD226B85CBE594271ADFB80E52" }, { "b" : "7F4A0DE58000", "path" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7F4A0DB7B000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30", "elfType" : 3, "buildId" : "0C029B406758FAFD5EC0E9B2C1F81D8D5FFC9D26" }, { "b" : "7F4A0D737000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7F4A0D42E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "05451CB4D66C321691F64F253880B7CE5B8812A6" }, { "b" : "7F4A0D1DD000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "ADF6F41F223D42193165FA0C55871F02D915FB19" }, { "b" : "7F4A0CFCE000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "6B9F4061A1D44813A54DA4DBB0088F529D8D78EA" }, { "b" : "7F4A0CDB3000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7F4A0CB69000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7F4A0C8FA000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "DED90CB11A7648E334E068A2585DB44E8CC07686" }, { "b" : "7F4A0C6DF000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "987FB5E5854E4E99A324899E4525818BF0BBCACA" }, { "b" : "7F4A0C476000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7F4A0C26E000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "0DBB8C21FC5D977098CA718BA2BFD6C4C21172E9" }, { "b" : "7F4A0C066000", "path" : "/usr/lib/x86_64-linux-gnu/libatomic.so.1", "elfType" : 3, "buildId" : "E314A4770C0CAB38C02E462CEFEDE517EE8C371F" }, { "b" : "7F4A0BE50000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F4A0BC33000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "84538E3C6CFCD5D4E3C0D2B6C3373F802915A498" }, { "b" : "7F4A0B86A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CBFA941A8EB7A11E4F90E81B66FCD5A820995D7C" }, { "b" : "7F4A0ED67000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A7D5A820B802049276B1FC26C8E845A3E194EB6B" }, { "b" : "7F4A0B650000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "340B7B463F981B8A0FB3451751F881DF1B0C2F74" }, { "b" : "7F4A0EF46000", "path" : "/lib/x86_64-linux-gnu/libudev.so.1", "elfType" : 3, "buildId" : "677E00F6578E7FD54267F859E744E43667001142" }, { "b" : "7F4A0B275000", "path" : "/usr/lib/x86_64-linux-gnu/libperl.so.5.22", "elfType" : 3, "buildId" : "9F15B66FC5A0C22FB46B2B2E91F3CE705857A05E" }, { "b" : "7F4A0B05C000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "C84AD91515AAAC66299EF9622D864E9DDD388C30" }, { "b" : "7F4A0AE1B000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "40315538B8F467FA9379C26806C2C2E5F75DF62A" }, { "b" : "7F4A0AAEB000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "B081530BA0EE8C7628C535957E776208328C3AD8" }, { "b" : "7F4A0A819000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7F4A0A5EA000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7F4A0A3E6000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7F4A0A1DB000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7F4A09FA8000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, { "b" : "7F4A09D8C000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B15D78D75542A4CAEC1DB7F7FB7723817E58D536" }, { "b" : "7F4A09B54000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "F8BB13675B60808A4AC354CC9CCF038EA19FD198" }, { "b" : "7F4A0994B000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "D72147C9E81DC821D2C5CA3A93757EE00DAC67EA" }, { "b" : "7F4A096C1000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "889221E5B7DFAA3D74CD8D55DAEF89DD962DAEB7" }, { "b" : "7F4A0941F000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "B7A0E46FB5107A9566AB0D448C865DB7BB1390BF" }, { "b" : "7F4A091EC000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "54027CA597C9AFC9D32A3D677857E5C8512C9FBF" }, { "b" : "7F4A08FD6000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "AE98B3C2FEC7AEDB3064E03BD8C1883FD12898D1" }, { "b" : "7F4A08D72000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7F4A08B5F000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "F5B37E248C55FD582ECD9B0240F96E8458F8E6F0" }, { "b" : "7F4A08929000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7F4A086F6000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7F4A08476000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7F4A08272000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7F4A08049000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "FEA2A08A986C1C954F7DED5A27616F0B4B18C6A2" }, { "b" : "7F4A07E3A000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "E5BC16E04FC04459D5AB8F045B5F7C7838B86027" }, { "b" : "7F4A07BEF000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "6B6EF07C8D0FA1CE7CCEE58043BDB6C347F25544" }, { "b" : "7F4A0791A000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7F4A07712000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x33) [0x563ae4d993e3]
 mongod(+0x474CEB5) [0x563ae4d98eb5]
 mongod(+0x474C8C9) [0x563ae4d988c9]
 libpthread.so.0(+0x11390) [0x7f4a0bc44390]
 libpthread.so.0(pthread_mutex_lock+0x4) [0x7f4a0bc3cd44]
 mongod(+0x1FB2393) [0x563ae25fe393]
 mongod(_ZNSt5mutex4lockEv+0x15) [0x563ae25ffb05]
 mongod(_ZNSt10lock_guardISt5mutexEC2ERS0_+0x23) [0x563ae265cd93]
 mongod(_ZN5mongo12CatalogCache27invalidateShardedCollectionERKNS_15NamespaceStringE+0x44) [0x563ae41538f4]
 mongod(_ZN5mongo12CatalogCache42getShardedCollectionRoutingInfoWithRefreshEPNS_16OperationContextERKNS_15NamespaceStringE+0x57) [0x563ae4153cf7]
 mongod(_ZN5mongo25SessionsCollectionSharded32_checkCacheForSessionsCollectionEPNS_16OperationContextE+0xD1) [0x563ae3ebcfb1]
 mongod(_ZN5mongo25SessionsCollectionSharded23setupSessionsCollectionEPNS_16OperationContextE+0x24) [0x563ae3ebd224]
 mongod(_ZN5mongo23LogicalSessionCacheImpl8_refreshEPNS_6ClientE+0x23C) [0x563ae3f3968c]
 mongod(_ZN5mongo23LogicalSessionCacheImpl16_periodicRefreshEPNS_6ClientE+0x6D) [0x563ae3f3abad]
 mongod(+0x38EFF10) [0x563ae3f3bf10]
 mongod(+0x38EFDC2) [0x563ae3f3bdc2]
 mongod(_ZNKSt8functionIFvPN5mongo6ClientEEEclES2_+0x64) [0x563ae3e72414]
 mongod(+0x381F3D0) [0x563ae3e6b3d0]
 mongod(+0x381EFCA) [0x563ae3e6afca]
 mongod(_ZNKSt8functionIFvSt10error_codeEEclES0_+0x67) [0x563ae4450c77]
 mongod(_ZN4asio6detail7binder1ISt8functionIFvSt10error_codeEES3_EclEv+0x2D) [0x563ae4450bed]
 mongod(_ZN4asio19asio_handler_invokeINS_6detail7binder1ISt8functionIFvSt10error_codeEES4_EEEEvRT_z+0x11) [0x563ae4450bb1]
 mongod(_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1ISt8functionIFvSt10error_codeEES5_EES7_EEvRT_RT0_+0x32) [0x563ae4450b92]
 mongod(_ZN4asio6detail19asio_handler_invokeINS0_7binder1ISt8functionIFvSt10error_codeEES4_EES6_S4_EEvRT_PNS2_IT0_T1_EE+0x1D) [0x563ae4450b4d]
 mongod(_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1ISt8functionIFvSt10error_codeEES5_EES8_EEvRT_RT0_+0x30) [0x563ae4450830]
 mongod(_ZN4asio6detail14strand_service8dispatchINS0_7binder1ISt8functionIFvSt10error_codeEES5_EEEEvRPNS1_11strand_implERT_+0x67) [0x563ae44505c7]
 mongod(_ZN4asio10io_context6strand8dispatchINS_6detail7binder1ISt8functionIFvSt10error_codeEES6_EEEENS_12async_resultINSt5decayIT_E4typeEFvvEE11return_typeEOSC_+0x45) [0x563ae44504a5]
 mongod(_ZN4asio6detail15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEclIS5_EEvRKT_+0x5F) [0x563ae445040f]
 mongod(_ZN4asio6detail7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES6_EclEv+0x1F) [0x563ae445039f]
 mongod(_ZN4asio6detail17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES7_EES9_EclEv+0x19) [0x563ae4450379]
 mongod(_ZN4asio19asio_handler_invokeINS_6detail17rewrapped_handlerINS1_7binder1INS1_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS1_26is_continuation_if_runningEEES8_EESA_EEEEvRT_z+0x11) [0x563ae4450351]
 mongod(_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail17rewrapped_handlerINS2_7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES9_EESB_EESB_EEvRT_RT0_+0x32) [0x563ae4450332]
 mongod(_ZN4asio6detail19asio_handler_invokeINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EESD_SA_EEvRT_PNS2_IT0_T1_EE+0x1D) [0x563ae44502ed]
 mongod(_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail17rewrapped_handlerINS2_7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES9_EESB_EESF_EEvRT_RT0_+0x30) [0x563ae444ffb0]
 mongod(_ZN4asio6detail12handler_workINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EENS_15system_executorEE8completeISE_EEvRT_RSE_+0x21) [0x563ae4451641]
 mongod(_ZN4asio6detail18completion_handlerINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES8_EESA_EEE11do_completeEPvPNS0_19scheduler_operationERKS8_m+0xF3) [0x563ae44501d3]
 mongod(_ZN4asio6detail14strand_service8dispatchINS0_17rewrapped_handlerINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES9_EESB_EEEEvRPNS1_11strand_implERT_+0x1AA) [0x563ae444fe8a]
 mongod(_ZN4asio10io_context6strand8dispatchINS_6detail17rewrapped_handlerINS3_7binder1INS3_15wrapped_handlerIS1_St8functionIFvSt10error_codeEENS3_26is_continuation_if_runningEEES8_EESA_EEEENS_12async_resultINSt5decayIT_E4typeEFvvEE11return_typeEOSH_+0x45) [0x563ae444fbd5]
 mongod(_ZN4asio6detail19asio_handler_invokeINS0_7binder1INS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEES7_EES5_S9_SA_EEvRT_PNS3_IT0_T1_T2_EE+0x69) [0x563ae444fb39]
 mongod(_ZN27asio_handler_invoke_helpers6invokeIN4asio6detail7binder1INS2_15wrapped_handlerINS1_10io_context6strandESt8functionIFvSt10error_codeEENS2_26is_continuation_if_runningEEES8_EESC_EEvRT_RT0_+0x30) [0x563ae444fac0]
 mongod(_ZN4asio6detail12handler_workINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEENS_15system_executorEE8completeINS0_7binder1ISA_S6_EEEEvRT_RSA_+0x21) [0x563ae444f901]
 mongod(_ZN4asio6detail12wait_handlerINS0_15wrapped_handlerINS_10io_context6strandESt8functionIFvSt10error_codeEENS0_26is_continuation_if_runningEEEE11do_completeEPvPNS0_19scheduler_operationERKS6_m+0xE5) [0x563ae444f675]
 mongod(_ZN4asio6detail19scheduler_operation8completeEPvRKSt10error_codem+0x47) [0x563ae4c7e7b7]
 mongod(_ZN4asio6detail9scheduler10do_run_oneERNS0_27conditionally_enabled_mutex11scoped_lockERNS0_21scheduler_thread_infoERKSt10error_code+0x208) [0x563ae4c6fed8]
 mongod(_ZN4asio6detail9scheduler3runERSt10error_code+0x106) [0x563ae4c65fd6]
 mongod(_ZN4asio10io_context3runERSt10error_code+0x21) [0x563ae4c66131]
 mongod(+0x3821356) [0x563ae3e6d356]
 mongod(+0x38211C5) [0x563ae3e6d1c5]
 mongod(+0x3821195) [0x563ae3e6d195]
 mongod(+0x3821089) [0x563ae3e6d089]
 mongod(+0x4915CF0) [0x563ae4f61cf0]
 libpthread.so.0(+0x76BA) [0x7f4a0bc3a6ba]
 libc.so.6(clone+0x6D) [0x7f4a0b97082d]
-----  END BACKTRACE  -----
Segmentation fault (core dumped)

Symbolized stack trace (bottom snipped):

 /home/eshamaharishi/code/mongo-master/src/mongo/util/stacktrace_posix.cpp:172:30: mongo::printStackTrace(std::ostream&)
 /home/eshamaharishi/code/mongo-master/src/mongo/util/signal_handlers_synchronous.cpp:180:5: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /home/eshamaharishi/code/mongo-master/src/mongo/util/signal_handlers_synchronous.cpp:276:5: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo_t*, void*)
 ??:0:0: ??
 ??:0:0: __GI___pthread_mutex_lock
 /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/x86_64-mongodb-linux/bits/gthr-default.h:748:12: __gthread_mutex_lock(pthread_mutex_t*)
 /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/mutex:135:17: std::mutex::lock()
 /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/mutex:386:19: std::lock_guard<std::mutex>::lock_guard(std::mutex&)
 /home/eshamaharishi/code/mongo-master/src/mongo/s/catalog_cache.cpp:363:35: mongo::CatalogCache::invalidateShardedCollection(mongo::NamespaceString const&)
 /home/eshamaharishi/code/mongo-master/src/mongo/s/catalog_cache.cpp:268:5: mongo::CatalogCache::getShardedCollectionRoutingInfoWithRefresh(mongo::OperationContext*, mongo::NamespaceString const&)
 /home/eshamaharishi/code/mongo-master/src/mongo/db/sessions_collection_sharded.cpp:60:50: mongo::SessionsCollectionSharded::_checkCacheForSessionsCollection(mongo::OperationContext*)
 /home/eshamaharishi/code/mongo-master/src/mongo/db/sessions_collection_sharded.cpp:75:12: mongo::SessionsCollectionSharded::setupSessionsCollection(mongo::OperationContext*)
 /home/eshamaharishi/code/mongo-master/src/mongo/db/logical_session_cache_impl.cpp:266:31: mongo::LogicalSessionCacheImpl::_refresh(mongo::Client*)
 /home/eshamaharishi/code/mongo-master/src/mongo/db/logical_session_cache_impl.cpp:164:9: mongo::LogicalSessionCacheImpl::_periodicRefresh(mongo::Client*)
 /home/eshamaharishi/code/mongo-master/src/mongo/db/logical_session_cache_impl.cpp:70:39: mongo::LogicalSessionCacheImpl::LogicalSessionCacheImpl(std::unique_ptr<mongo::ServiceLiason, std::default_delete<mongo::ServiceLiason> >, std::shared_ptr<mongo::SessionsCollection>, std::shared_ptr<mongo::TransactionReaper>, mongo::LogicalSessionCacheImpl::Options)::$_1::operator()(mongo::Client*) const
 /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:1871:2: std::_Function_handler<void (mongo::Client*), mongo::LogicalSessionCacheImpl::LogicalSessionCacheImpl(std::unique_ptr<mongo::ServiceLiason, std::default_delete<mongo::ServiceLiason> >, std::shared_ptr<mongo::SessionsCollection>, std::shared_ptr<mongo::TransactionReaper>, mongo::LogicalSessionCacheImpl::Options)::$_1>::_M_invoke(std::_Any_data const&, mongo::Client*&&)
 /opt/mongodbtoolchain/v2/bin/../lib/gcc/x86_64-mongodb-linux/5.4.0/../../../../include/c++/5.4.0/functional:2267:14: std::function<void (mongo::Client*)>::operator()(mongo::Client*) const
 /home/eshamaharishi/code/mongo-master/src/mongo/util/periodic_runner_asio.cpp:96:13: mongo::PeriodicRunnerASIO::_scheduleJob(std::weak_ptr<mongo::PeriodicRunnerASIO::PeriodicJobASIO>, bool)::$_0::operator()(std::error_code)

Comment by Esha Maharishi (Inactive) [ 18/Apr/18 ]

I'm interested in what the steps taken by mongo-orchestration and the python test are, since it takes a bit of manipulation to trigger the invalid access in SERVER-32677.

It'd be useful to know whether it's the same manipulation, or if this can be triggered another way.

Comment by Esha Maharishi (Inactive) [ 18/Apr/18 ]

This seems almost identical to a known issue in SERVER-32677, see the stack trace commented there.

Comment by Bruce Lucas (Inactive) [ 18/Apr/18 ]

Looks like the next-to-top frame points to this line in invalidateShardedCollection:

    stdx::lock_guard<stdx::mutex> lg(_mutex);

The top frame is an access violation in Mtx_destroy. It seems odd that code associated with lock_guard would call Mtx_destroy.

Comment by Shane Harvey [ 17/Apr/18 ]

What's happening as far as I can tell:

  1. Mongo-orchestration successfully starts a sharded cluster with auth/ssl with 2 shards (single member replica sets), 1 config server, and 2 mongoses.
  2. Mongo-orchestration adds the bob@admin user to each shard and to the mongos.
  3. PyMongo test suite runs but cannot connect to the mongos on port 27017: SSL handshake failed: [Errno 10054] An existing connection was forcibly closed by the remote host
  4. Mongo orchestration starts failing to authenticate to the shard on port 27217.

      2018-04-17T20:33:46.689+0000 I NETWORK  [listener] connection accepted from 127.0.0.1:49542 #6 (3 connections now open)
      2018-04-17T20:33:46.716+0000 I NETWORK  [conn6] received client metadata from 127.0.0.1:49542 conn6: { driver: { name: "PyMongo", version: "3.5.1" }, os: { type: "Windows", name: "Windows 2008ServerR2", architecture: "AMD64", version: "6.1.7601-SP1" }, platform: "CPython 2.7.14.final.0" }
      2018-04-17T20:33:46.717+0000 I ACCESS   [conn6] SASL SCRAM-SHA-1 authentication failed for bob on admin from client 127.0.0.1:49542 ; UserNotFound: Could not find user bob@admin
      

  5. Sometime later the mongod on port 27217 fails.

Mongo-orchestration log file: mongo-orchestration.log
mongos 27017 log file: mongos-27017.log
config server log file: mongod-1026.log
shard port 27217 log file: mongod-27217.log
shard port 27218 log file: mongod-27218.log

Comment by Esha Maharishi (Inactive) [ 17/Apr/18 ]

shane.harvey, can you include what the driver test does to help with repro'ing this?

Generated at Thu Feb 08 04:36:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.