[SERVER-28530] SSLThreadInfo invokes constructor in destructor Created: 28/Mar/17  Updated: 07/Sep/17  Resolved: 10/Apr/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 3.4.4, 3.5.6

Type: Task Priority: Major - P3
Reporter: Spencer Jackson Assignee: Spencer Jackson
Resolution: Done Votes: 0
Labels: bkp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Duplicate
is duplicated by SERVER-28660 memory leak of mongodb database with ... Closed
Related
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.4, v3.2, v3.0
Sprint: Platforms 2017-04-17
Participants:

 Description   

SSLThreadInfo is stored in a boost::thread_specific_ptr. When the thread shuts down, thread_specific_ptr calls SSLThreadInfo's deconstructor. SSLThreadInfo::~SSLThreadInfo calls 'ERR_remove_state(0)'. In the below stacktrace, ERR_remove_state calls CRYPTO_THREADID_current, which calls our callback to obtain a thread ID OpenSSL uses to track resources. Our callback uses the SSLThreadInfo stored in the boost::thread_specific_ptr. Because the active ptr doesn't exist(because it's being destroyed), our callback makes a new SSLThreadInfo, and allocates a new thread ID.

This could impact OpenSSL's ability to correctly manage thread local memory.

(gdb) bt
#0  0x00007ffff50faea0 in raise () from /usr/lib/libpthread.so.0
#1  0x000055555a4a42ab in mongo::breakpoint () at src/mongo/util/debugger.cpp:75
#2  0x000055555a667f18 in mongo::(anonymous namespace)::myTerminate () at src/mongo/util/signal_handlers_synchronous.cpp:229
#3  0x00007ffff53962a6 in __cxxabiv1::__terminate (handler=<optimized out>) at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007ffff53962f1 in std::terminate () at /build/gcc/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x000055555a5658f8 in mongo::(anonymous namespace)::SSLThreadInfo::SSLThreadInfo (this=0x602000140010) at src/mongo/util/net/ssl_manager.cpp:169
#6  0x000055555a560f25 in mongo::(anonymous namespace)::SSLThreadInfo::get () at src/mongo/util/net/ssl_manager.cpp:199
#7  0x000055555a560c99 in mongo::(anonymous namespace)::_ssl_id_callback () at src/mongo/util/net/ssl_manager.cpp:217
#8  0x00007ffff6786a72 in CRYPTO_THREADID_current () from /usr/lib/libcrypto.so.1.0.0
#9  0x00007ffff685f768 in ERR_remove_thread_state () from /usr/lib/libcrypto.so.1.0.0
#10 0x000055555a576710 in mongo::(anonymous namespace)::SSLThreadInfo::~SSLThreadInfo (this=0x60200013f650) at src/mongo/util/net/ssl_manager.cpp:174
#11 0x000055555a5766db in boost::thread_specific_ptr<mongo::(anonymous namespace)::SSLThreadInfo>::delete_data::operator() (this=0x602000002ad0, data=0x60200013f650) at src/third_party/boost-1.60.0/boost/thread/tss.hpp:42
#12 0x000055555b00f258 in boost::detail::(anonymous namespace)::tls_destructor (data=0x6130000c0a00) at src/third_party/boost-1.60.0/libs/thread/src/pthread/thread.cpp:108
#13 0x00007ffff50ef429 in __nptl_deallocate_tsd.part.4 () from /usr/lib/libpthread.so.0
#14 0x00007ffff50f0498 in start_thread () from /usr/lib/libpthread.so.0
#15 0x00007ffff4c1a54f in clone () from /usr/lib/libc.so.6



 Comments   
Comment by Githook User [ 13/Apr/17 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-28530 Prevent SSLThreadInfo destruction construction cycle

(cherry picked from commit 169f0f1178347f367d56b8ef8ec608b11ca71307)
Branch: v3.4
https://github.com/mongodb/mongo/commit/c741cb337cc22378dc8acbe34de90d7011f35551

Comment by Githook User [ 10/Apr/17 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-28530 Prevent SSLThreadInfo destruction construction cycle
Branch: master
https://github.com/mongodb/mongo/commit/169f0f1178347f367d56b8ef8ec608b11ca71307

Comment by Misha Tyulenev [ 29/Mar/17 ]

Thanks spencer.jackson one of the stacks from HELP-3980 that leaks memory matches:

"stack677":{"activeBytes":81526784,"stack":{
"0":"operator new[]",
"1":"boost::detail::make_external_thread_data",
"2":"boost::detail::add_new_tss_node",
"3":"boost::detail::set_tss_data",
"4":"0x7f5ece904ea4",
"5":"CRYPTO_THREADID_current",
"6":"ERR_remove_thread_state",
"7":"0x7f5ece9048e8","8":"0x7f5eceaf8337","9":"0x7f5ecca81a69","10":"clone"}}

Generated at Thu Feb 08 04:18:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.