[SERVER-26369] Crash on start-up for shardsrvr when enabling SSL with encrypted PEM for a cluster Created: 28/Sep/16  Updated: 21/Feb/20  Resolved: 08/Nov/16

Status: Closed
Project: Core Server
Component/s: Security, Sharding
Affects Version/s: 3.3.12
Fix Version/s: 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Andy Schwerin Assignee: Spencer Jackson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OSX, Amazon Linux AMI release 2016.03, apparently not Ubuntu 16.04


Attachments: File shard0.tar.gz    
Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

start 3.3.12 enterprise with the following configuration, using PEM files from the jstests/libs directory of the mongodb/mongo repo, the error reproduces at least on OS X.
/path/to/mongod \
--dbpath shard0 \
--shardsvr \
--replSet shard0 \
--sslMode allowSSL \
--sslCAFile mongo/jstests/libs/ca.pem \
--sslPEMKeyFile mongo/jstests/libs/password_protected.pem \
--sslPEMKeyPassword qwerty

You can also reduce it from scratch, by creating a sharded cluster with a 1-node replica set config server, and a 1-node replica set shard without SSL. Then, shut everything down, and just try to start the shard server as described above. It should be easy to write a repro js script for this, but I haven't had time, yet.

Sprint: Platforms 2016-10-10, Security 2016-11-21
Participants:

 Description   

When starting a shard server with --sslMode=allowSSL and supplying a password-encrypted PEM file, the shard server crashes at start-up, while performing sharding state initialization. The stack trace indicates possible trouble creating or destroying an SSL manager:

mongo/util/net/ssl_manager.h:74:8: mongo::SSLConfiguration::~SSLConfiguration()
mongo/util/net/ssl_manager.h:74:0: mongo::SSLConfiguration::~SSLConfiguration()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
third_party/asio-asio-1-11-0/asio/include/asio/ssl/impl/context.ipp:200:7: asio::ssl::context::~context()
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2459:13: std::__1::default_delete<asio::ssl::context>::operator()(asio::ssl::con
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2658:0: std::__1::unique_ptr<asio::ssl::context, std::__1::default_delete<asio::
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<asio::ssl::context, std::__1::default_delete<asio::
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<asio::ssl::context, std::__1::default_delete<asio::
mongo/util/net/asio_ssl_context.h:45:0: mongo::ASIOSSLContext::~ASIOSSLContext()
mongo/util/net/asio_ssl_context.h:45:0: mongo::ASIOSSLContext::~ASIOSSLContext()
mongo/util/net/asio_message_port.cpp:58:0: mongo::(anonymous namespace)::ASIOSSLContextPair::~ASIOSSLContextPair()
mongo/util/net/asio_message_port.cpp:58:0: mongo::(anonymous namespace)::ASIOSSLContextPair::~ASIOSSLContextPair()
mongo/util/decoration_registry.h:118:0: void mongo::DecorationRegistry::destructAt<mongo::(anonymous namespace)::ASIOSSLContextPair>(void*)
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/functional:1793:12: std::__1::function<void (void*)>::operator()(void*) const
mongo/util/decoration_registry.cpp:75:0: mongo::DecorationRegistry::destruct(mongo::DecorationContainer*) const
mongo/util/decoration_container.cpp:44:5: mongo::DecorationContainer::~DecorationContainer()
mongo/util/decoration_container.cpp:43:0: mongo::DecorationContainer::~DecorationContainer()
mongo/util/decorable.h:110:6: mongo::Decorable<mongo::SSLManagerInterface>::~Decorable()
mongo/util/net/ssl_manager.cpp:502:0: mongo::SSLManagerInterface::~SSLManagerInterface()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
mongo/util/net/ssl_manager.cpp:206:0: mongo::(anonymous namespace)::SSLManager::~SSLManager()
third_party/asio-asio-1-11-0/asio/include/asio/ssl/impl/context.ipp:200:7: asio::ssl::context::~context()
mongo/executor/async_secure_stream_factory.h:46:7: mongo::executor::AsyncSecureStreamFactory::~AsyncSecureStreamFactory()
mongo/executor/async_secure_stream_factory.h:46:0: mongo::executor::AsyncSecureStreamFactory::~AsyncSecureStreamFactory()
mongo/executor/async_secure_stream_factory.h:46:0: mongo::executor::AsyncSecureStreamFactory::~AsyncSecureStreamFactory()
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2459:13: std::__1::default_delete<mongo::executor::AsyncStreamFactoryInterface>:
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2658:0: std::__1::unique_ptr<mongo::executor::AsyncStreamFactoryInterface, std::
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::AsyncStreamFactoryInterface, std::
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::AsyncStreamFactoryInterface, std::
mongo/executor/network_interface_asio.h:95:0: mongo::executor::NetworkInterfaceASIO::~NetworkInterfaceASIO()
mongo/executor/network_interface_asio.h:95:7: mongo::executor::NetworkInterfaceASIO::~NetworkInterfaceASIO()
mongo/executor/network_interface_asio.h:95:0: mongo::executor::NetworkInterfaceASIO::~NetworkInterfaceASIO()
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2459:13: std::__1::default_delete<mongo::executor::NetworkInterface>::operator()
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2658:0: std::__1::unique_ptr<mongo::executor::NetworkInterface, std::__1::defaul
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::NetworkInterface, std::__1::defaul
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::NetworkInterface, std::__1::defaul
mongo/executor/thread_pool_task_executor.cpp:129:0: mongo::executor::ThreadPoolTaskExecutor::~ThreadPoolTaskExecutor()
mongo/executor/thread_pool_task_executor.cpp:129:51: mongo::executor::ThreadPoolTaskExecutor::~ThreadPoolTaskExecutor()
mongo/executor/thread_pool_task_executor.cpp:129:0: mongo::executor::ThreadPoolTaskExecutor::~ThreadPoolTaskExecutor()
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2459:13: std::__1::default_delete<mongo::executor::TaskExecutor>::operator()(mon
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2658:0: std::__1::unique_ptr<mongo::executor::TaskExecutor, std::__1::default_de
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::TaskExecutor, std::__1::default_de
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/memory:2626:0: std::__1::unique_ptr<mongo::executor::TaskExecutor, std::__1::default_de
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__functional_base:415:0: decltype(std::__1::forward<mongo::initializeGlobalShardingStat
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__functional_base:430:0: std::__1::unique_ptr<mongo::ShardingCatalogManager, std::__1::
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/functional:1407:0: std::__1::__function::__func<mongo::initializeGlobalShardingStateFor
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/functional:1793:12: std::__1::function<std::__1::unique_ptr<mongo::ShardingCatalogManag
mongo/s/sharding_initialization.cpp:163:0: mongo::initializeGlobalShardingState(mongo::OperationContext*, mongo::ConnectionString const&, mongo::Strin
mongo/db/s/sharding_initialization_mongod.cpp:83:12: mongo::initializeGlobalShardingStateForMongod(mongo::OperationContext*, mongo::ConnectionString c
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__functional_base:415:12: decltype(std::__1::forward<mongo::Status (*&)(mongo::Operatio
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/__functional_base:430:0: mongo::Status std::__1::__invoke_void_return_wrapper<mongo::St
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/functional:1407:0: std::__1::__function::__func<mongo::Status (*)(mongo::OperationConte
 /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1/functional:1793:12: std::__1::function<mongo::Status (mongo::OperationContext*, mongo::
mongo/db/s/sharding_state.cpp:454:0: mongo::ShardingState::initializeFromShardIdentity(mongo::OperationContext*, mongo::ShardIdentityType const&)
mongo/db/s/sharding_state.cpp:653:27: mongo::ShardingState::initializeShardingAwarenessIfNeeded(mongo::OperationContext*)
mongo/db/db.cpp:721:5: mongo::_initAndListen(int)
mongo/db/db.cpp:793:16: mongo::initAndListen(int)
mongo/db/db.cpp:1143:25: mongoDbMain(int, char**, char**)
mongo/db/db.cpp:836:0: main



 Comments   
Comment by Githook User [ 21/Feb/20 ]

Author:

{'name': 'Randolph Tan', 'username': 'renctan', 'email': 'randolph@10gen.com'}

Message: SERVER-46301 Add requires_persistence tag to SERVER-26369.js
Branch: master
https://github.com/mongodb/mongo/commit/39c6f724daaaf2f0d0f16d815f9fe281dd7dbba9

Comment by Githook User [ 08/Nov/16 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-26369: Fix shard server crash with encrypted PEMKeyFiles
Branch: master
https://github.com/mongodb/mongo/commit/3220495083b0d678578a76591f54ee1d7a5ec5df

Comment by Andy Schwerin [ 28/Sep/16 ]

spencer.jackson, after you reproduce this, I recommend writing a js test that you can run in a patch build against all the builders, to see where it reproduces.

Generated at Thu Feb 08 04:11:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.