[SERVER-29951] mongodb-3.4.4 crashed: Invalid access at address Created: 01/Jul/17  Updated: 29/Jul/17  Resolved: 07/Jul/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Peter Assignee: Andrew Morrow (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hi. mongodb-3.4.4 crashed here. Stack trace is the following:

2017-07-01T09:55:37.083+0300 F -        [thread1] Invalid access at address: 0x7fb89e9add13
2017-07-01T09:55:37.243+0300 F -        [thread1] Got signal: 11 (Segmentation fault).
 
 0x55cb10181da6 0x55cb10180e6c 0x55cb101814ee 0x7fb8c1546640 0x7fb8c153cdd1 0x55cb100e1be3 0x55cb0f43c771 0x55cb100e577f 0x55cb100e6d4b 0x55cb100e38e3 0x55cb1011eb55 0x55cb1011f361 0x7fb8c1d59d2d 0x7fb8c153c5e0 0x7fb8c127fffd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55CB0ECC9000","o":"14B8DA6","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55CB0ECC9000","o":"14B7E6C"},{"b":"55CB0ECC9000","o":"14B84EE"},{"b":"7FB8C1535000","o":"11640"},{"b":"7FB8C1535000","o":"7DD1","s":"pthread_create"},{"b":"55CB0ECC9000","o":"1418BE3","s":"_ZN5mongo37launchWrappedServiceEntryWorkerThreadESt10shared_ptrINS_9transport7SessionEESt8functionIFvRKS3_EE"},{"b":"55CB0ECC9000","o":"773771","s":"_ZN5mongo23ServiceEntryPointMongod12startSessionESt10shared_ptrINS_9transport7SessionEE"},{"b":"55CB0ECC9000","o":"141C77F","s":"_ZN5mongo9transport20TransportLayerLegacy20_handleNewConnectionESt10unique_ptrINS_21AbstractMessagingPortESt14default_deleteIS3_EE"},{"b":"55CB0ECC9000","o":"141DD4B","s":"_ZNSt17_Function_handlerIFvSt10unique_ptrIN5mongo21AbstractMessagingPortESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_9transport20TransportLayerLegacyEFvS5_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataOS5_"},{"b":"55CB0ECC9000","o":"141A8E3","s":"_ZN5mongo9transport20TransportLayerLegacy14ListenerLegacy8acceptedESt10unique_ptrINS_21AbstractMessagingPortESt14default_deleteIS4_EE"},{"b":"55CB0ECC9000","o":"1455B55","s":"_ZN5mongo8Listener9_acceptedERKSt10shared_ptrINS_6SocketEEx"},{"b":"55CB0ECC9000","o":"1456361","s":"_ZN5mongo8Listener13initAndListenEv"},{"b":"7FB8C1C6D000","o":"ECD2D"},{"b":"7FB8C1535000","o":"75E0"},{"b":"7FB8C1192000","o":"EDFFD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.4", "gitVersion" : "888390515874a9debd1b6c5d36559ca86b44babd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.16-gentoo", "version" : "#2 SMP Tue Sep 6 07:59:44 MSK 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "55CB0ECC9000", "elfType" : 3, "buildId" : "FA5F2E82A54638A6190E6FA6BC93AC792F8B4596" }, { "b" : "7FFC43DD9000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "02D07281CA55F8126FDAB95CBF301757774DCCB0" }, { "b" : "7FB8C421D000", "path" : "/usr/lib64/libstemmer.so.0d", "elfType" : 3 }, { "b" : "7FB8C3F9E000", "path" : "/usr/lib64/libyaml-cpp.so.0.5", "elfType" : 3 }, { "b" : "7FB8C3D18000", "path" : "/usr/lib64/libboost_program_options.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C3AFD000", "path" : "/usr/lib64/libboost_filesystem.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C38D2000", "path" : "/usr/lib64/libboost_thread.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C36CE000", "path" : "/usr/lib64/libboost_system.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C34C7000", "path" : "/usr/lib64/libboost_chrono.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C31D4000", "path" : "/usr/lib64/libboost_regex.so.1.62.0", "elfType" : 3 }, { "b" : "7FB8C2FBB000", "path" : "/lib64/libz.so.1", "elfType" : 3 }, { "b" : "7FB8C2DB1000", "path" : "/usr/lib64/libsnappy.so.1", "elfType" : 3 }, { "b" : "7FB8C2BA6000", "path" : "/usr/lib64/libpcrecpp.so.0", "elfType" : 3 }, { "b" : "7FB8C292D000", "path" : "/usr/lib64/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FB8C24AC000", "path" : "/usr/lib64/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FB8C22A4000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FB8C20A0000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FB8C1C6D000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FB8C1969000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FB8C1752000", "path" : "/usr/lib/gcc/x86_64-pc-linux-gnu/6.3.0/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FB8C1535000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FB8C1192000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FB8C0F4D000", "path" : "/lib64/libpcre.so.1", "elfType" : 3 }, { "b" : "7FB8C4473000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x46) [0x55cb10181da6]
 mongod(+0x14B7E6C) [0x55cb10180e6c]
 mongod(+0x14B84EE) [0x55cb101814ee]
 libpthread.so.0(+0x11640) [0x7fb8c1546640]
 libpthread.so.0(pthread_create+0x521) [0x7fb8c153cdd1]
 mongod(_ZN5mongo37launchWrappedServiceEntryWorkerThreadESt10shared_ptrINS_9transport7SessionEESt8functionIFvRKS3_EE+0x133) [0x55cb100e1be3]
 mongod(_ZN5mongo23ServiceEntryPointMongod12startSessionESt10shared_ptrINS_9transport7SessionEE+0x71) [0x55cb0f43c771]
 mongod(_ZN5mongo9transport20TransportLayerLegacy20_handleNewConnectionESt10unique_ptrINS_21AbstractMessagingPortESt14default_deleteIS3_EE+0x29F) [0x55cb100e577f]
 mongod(_ZNSt17_Function_handlerIFvSt10unique_ptrIN5mongo21AbstractMessagingPortESt14default_deleteIS2_EEESt5_BindIFSt7_Mem_fnIMNS1_9transport20TransportLayerLegacyEFvS5_EEPSA_St12_PlaceholderILi1EEEEE9_M_invokeERKSt9_Any_dataOS5_+0x5B) [0x55cb100e6d4b]
 mongod(_ZN5mongo9transport20TransportLayerLegacy14ListenerLegacy8acceptedESt10unique_ptrINS_21AbstractMessagingPortESt14default_deleteIS4_EE+0x53) [0x55cb100e38e3]
 mongod(_ZN5mongo8Listener9_acceptedERKSt10shared_ptrINS_6SocketEEx+0x345) [0x55cb1011eb55]
 mongod(_ZN5mongo8Listener13initAndListenEv+0x621) [0x55cb1011f361]
 libstdc++.so.6(+0xECD2D) [0x7fb8c1d59d2d]
 libpthread.so.0(+0x75E0) [0x7fb8c153c5e0]
 libc.so.6(clone+0x6D) [0x7fb8c127fffd]
-----  END BACKTRACE  -----

Before crash there was a lot of lines in log:

2017-07-01T09:55:34.779+0300 I -        [conn299001] AssertionException handling request, closing client connection: 6 socket exception [SEND_ERROR] for 172.16.11.43:60425



 Comments   
Comment by Andrew Morrow (Inactive) [ 07/Jul/17 ]

Well, I can tell a fair amount from this stack, but not enough to know exactly what went wrong. It looks like we crashed inside pthread_create. There have been a few issues over the years where a failed call to pthread_create would lead to a subsequent crash, but this doesn't really meet those patterns - typically there would have been an error message logged. I also wouldn't expect pthread_create to crash if this was a failure to acquire necessary resources - I'd expect it to fail with an error, and we do check for that in 3.4.4. Please do re-build with symbols, and set any necessary ulimits or system policies to enable core dumps for the server process.

In the meanwhile, I'm going to close this ticket as 'Cannot Reproduce'. However, if you do happen to reproduce it, feel free to re-open the ticket (or just comment on it; I'll be watching) and I will take a closer look. Let me know if there is anything else I can do to help.

Comment by Peter [ 07/Jul/17 ]

Hi, Andrew. This binary was self built and unfortunately without symbols. This stack trace is all I have. If problem is not clear, let's close this bug for a moment. This crash happen only once. Meanwhile, I'll build newer mongodb with symbols and gather core file as well.

Comment by Andrew Morrow (Inactive) [ 07/Jul/17 ]

Hi pva -

Gentoo isn't one of our supported platforms for which we produce binary builds, so we are going to need some additional information to make progress here. We can't symbolize this stack trace (to get a line number) because we don't have the binaries you ran, so the first question is whether or not you happen to have a core dump from this crash again your current binary. If so, it would be very helpful if you could load that binary and core in GDB and give us the output of thread apply all bt. It would also be useful if you could upload the logs from mongodb showing what happened leading up to this crash.

Is the crash reproducible on your system? If so, what steps lead to it happening?

Finally, if you could provide some background on where you obtained the binaries, that would be useful.

Generated at Thu Feb 08 04:22:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.