[SERVER-28069] Mongos crashes Created: 22/Feb/17  Updated: 24/Aug/17  Resolved: 17/Jul/17

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 3.4.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Edik Mkoyan Assignee: Jonathan Reams
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-27863 Reschedule early alarms in NetworkInt... Closed
Related
is related to SERVER-23199 Timer logic in NetworkInterfaceASIO i... Closed
Operating System: Linux
Participants:

 Description   

Mongos crashes with the following log

BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F7803F4D000","o":"BA46E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F7803F4D000","o":"BA37D9"},{"b":"7F7803F4D000","o":"BA3CBD"},{"b":"7F7802BA7000","o":"10330"},{"b":"7F78027E2000","o":"36C37","s":"gsignal"},{"b":"7F78027E2000","o":"3A028","s":"abort"},{"b":"7F7803F4D000","o":"3DC7BE","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F7803F4D000","o":"8F88B8"},{"b":"7F7803F4D000","o":"8CFB15"},{"b":"7F7803F4D000","o":"C82294","s":"_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code"},{"b":"7F7803F4D000","o":"C82491","s":"_ZN4asio6detail9scheduler3runERSt10error_code"},{"b":"7F7803F4D000","o":"8CEC1F"},{"b":"7F7803F4D000","o":"F82960"},{"b":"7F7802BA7000","o":"8184"},{"b":"7F78027E2000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-33-generic", "version" : "#38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "7F7803F4D000", "elfType" : 3, "buildId" : "149E88146384D65E439E13324AF97C149CF35650" }, { "b" : "7FFFE0533000", "elfType" : 3, "buildId" : "3AA714083E1D7D719D6C5A7B1A4A8D3649727CB8" }, { "b" : "7F7803AC9000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "48A664AE6B0B4918A3EB0156C6364C4F084232FD" }, { "b" : "7F78036ED000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "6B8997EA892A7FF37AC8CAA8F239D595251889BB" }, { "b" : "7F78034E5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F78032E1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F7802FDB000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F7802DC5000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F7802BA7000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F78027E2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F7803D28000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" }, { "b" : "7F78025D7000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "A0BDDA36D5AD1390D4DF158D7AD7CC866E534C28" }, { "b" : "7F78023D1000", "path" : "/lib/x86_64-linux-gnu/libnss_dns.so.2", "elfType" : 3, "buildId" : "A5721A2FACA0CB423F6C851DF8B7A96C52E94691" }, { "b" : "7F78021B6000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "FE705AA7E3872B43EA9E99FAF1897F7F85FC0AC9" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7f7804af16e1]
 mongos(+0xBA37D9) [0x7f7804af07d9]
 mongos(+0xBA3CBD) [0x7f7804af0cbd]
 libpthread.so.0(+0x10330) [0x7f7802bb7330]
 libc.so.6(gsignal+0x37) [0x7f7802818c37]
 libc.so.6(abort+0x148) [0x7f780281c028]
 mongos(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f78043297be]
 mongos(+0x8F88B8) [0x7f78048458b8]
 mongos(+0x8CFB15) [0x7f780481cb15]
 mongos(_ZN4asio6detail9scheduler10do_run_oneERNS0_11scoped_lockINS0_11posix_mutexEEERNS0_21scheduler_thread_infoERKSt10error_code+0x324) [0x7f7804bcf294]
 mongos(_ZN4asio6detail9scheduler3runERSt10error_code+0xD1) [0x7f7804bcf491]
 mongos(+0x8CEC1F) [0x7f780481bc1f]
 mongos(+0xF82960) [0x7f7804ecf960]
 libpthread.so.0(+0x8184) [0x7f7802baf184]
 libc.so.6(clone+0x6D) [0x7f78028dc37d]
-----  END BACKTRACE  -----



 Comments   
Comment by Ramon Fernandez Marina [ 17/Jul/17 ]

edikmkoyan, we haven't heard back from you in a while, so we're closing this ticket. If this is still happening for you after upgrading to 3.4.3 or newer please let us know so we can reopen the ticket (or feel free to open a new one).

Regards,
Ramón.

Comment by Jonathan Reams [ 26/Jun/17 ]

edikmkoyan, this should have been fixed in 3.4.3 by SERVER-27863. Is this still an issue for you?

Comment by Kaloian Manassiev [ 02/Mar/17 ]

Thank you anonymous.user. The invariant in question is invariant(now() >= when) and is due to a race in the timeout logic.

Passing on to the platforms team. We fixed a related issue (SERVER-23199) some time ago, but it seems that it is still happening in 3.4.2.

Comment by Kelsey Schubert [ 01/Mar/17 ]

parsed with mongodb-linux-x86_64-ubuntu1404-debugsymbols-3.4.2:

 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/util/signal_handlers_synchronous.cpp:178:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/util/signal_handlers_synchronous.cpp:235:0: mongo::(anonymous namespace)::abruptQuit(int)
 ??:0:0: ??
 ??:0:0: ??
 ??:0:0: ??
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/util/assert_util.cpp:154:0: mongo::invariantFailed(char const*, char const*, unsigned int)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/executor/thread_pool_task_executor.cpp:295:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::executor::ThreadPoolTaskExecutor::scheduleWorkAt(mongo::Date_t, std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> const&)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/executor/network_interface_asio.cpp:482:0: operator()
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/bind_handler.hpp:62:0: operator()
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/handler_invoke_hook.hpp:68:0: asio_handler_invoke<asio::detail::binder1<mongo::executor::NetworkInterfaceASIO::setAlarm(mongo::Date_t, const std::function<void()>&)::<lambda(std::error_code)>, std::error_code> >
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/handler_invoke_helpers.hpp:37:0: invoke<asio::detail::binder1<mongo::executor::NetworkInterfaceASIO::setAlarm(mongo::Date_t, const std::function<void()>&)::<lambda(std::error_code)>, std::error_code>, mongo::executor::NetworkInterfaceASIO::setAlarm(mongo::Date_t, const std::function<void()>&)::<lambda(std::error_code)> >
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/handler_work.hpp:81:0: complete<asio::detail::binder1<mongo::executor::NetworkInterfaceASIO::setAlarm(mongo::Date_t, const std::function<void()>&)::<lambda(std::error_code)>, std::error_code> >
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/wait_handler.hpp:71:0: asio::detail::wait_handler<mongo::executor::NetworkInterfaceASIO::setAlarm(mongo::Date_t, std::function<void ()> const&)::'lambda'(std::error_code)>::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/scheduler_operation.hpp:39:0: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/impl/scheduler.ipp:369:0: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/third_party/asio-asio-1-11-0/asio/include/asio/detail/impl/scheduler.ipp:146:0: asio::detail::scheduler::run(std::error_code&)
 /data/mci/4f5e6ace89cf7d17b1e5bb9701ff90dc/src/src/mongo/executor/network_interface_asio.cpp:159:0: mongo::executor::NetworkInterfaceASIO::startup()::'lambda'()::operator()() const (.constprop.359)
 /data/mci/d00a0293fbef024c26e63225fb9a70f5/toolchain-builder/build-gcc-v2.sh-nKN/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine
 ??:0:0: ??
 ??:0:0: ??

Comment by Kaloian Manassiev [ 22/Feb/17 ]

Hi edikmkoyan,

Can you please attach the complete mongos log file and let us know which distribution of MongoDB server you are using? I.e., the OS and the package version that you installed. If you have the download link that would also be great.

Best regards,
-Kal.

Generated at Thu Feb 08 04:17:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.