[SERVER-30910] Mongo getting Fatal Assertion 16782 when it rotates its logs Created: 31/Aug/17  Updated: 01/Sep/17  Resolved: 01/Sep/17

Status: Closed
Project: Core Server
Component/s: Logging
Affects Version/s: 3.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dennis Andersen Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Had this happen two days in a row:

2017-08-31T06:45:10.013-0700 I -        [signalProcessingThread] Fatal Assertion 16782
2017-08-31T06:45:10.048-0700 I CONTROL  [signalProcessingThread] 
 0xf3f529 0xedfd81 0xec5191 0xf3e76f 0xf8caf4 0x7faabd67a182 0x7faabc14247d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B3F529"},{"b":"400000","o":"ADFD81"},{"b":"400000","o":"AC5191"},{"b":"400000","o":"B3E76F"},{"b":"400000","o":"B8CAF4"},{"b":"7FAABD672000","o":"8182"},{"b":"7FAABC048000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.0.0", "gitVersion" : "a841fd6394365954886924a35076691b4d149168", "uname" : { "sysname" : "Linux", "release" : "3.13.0-51-generic", "version" : "#84-Ubuntu SMP Wed Apr 15 12:08:34 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "5F365548FE8312E027817D98FEECA3A18F0F60FB" }, { "b" : "7FFC904B3000", "elfType" : 3, "buildId" : "F8E01F1BCB9F27FF04FC2B8C9454AD8EACB7B37C" }, { "b" : "7FAABD672000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FAABD414000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "661B6E1F8640A26B5E2D2F923BBF3CF11A923554" }, { "b" : "7FAABD039000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "78636CCB09F0CBA8B8B06CCFD0B96F610F2645C1" }, { "b" : "7FAABCE31000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FAABCC2D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FAABC929000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "19EFDDAB11B3BF5C71570078C59F91CF6592CE9E" }, { "b" : "7FAABC623000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FAABC40D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FAABC048000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FAABD890000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf3f529]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xedfd81]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xec5191]
 mongod(+0xB3E76F) [0xf3e76f]
 mongod(+0xB8CAF4) [0xf8caf4]
 libpthread.so.0(+0x8182) [0x7faabd67a182]
 libc.so.6(clone+0x6D) [0x7faabc14247d]
-----  END BACKTRACE  -----
2017-08-31T06:45:10.048-0700 I -        [signalProcessingThread] 
 
***aborting after fassert() failure

It didn't start up the after the first time with this error:

2017-08-30T07:42:54.008-0700 I STORAGE  [initandlisten] exception in initAndListen std::exception: boost::filesystem::remove: Permission denied: "/var/lib/mongo/_tmp/", terminating

That folder was empty and looked like:
drwxr-xr-x  2 mongodb nogroup       4096 Aug 29 00:52 _tmp/

The second time we got 16782, it started without any problems at all.

There is plenty of free disk on the machine, and to my knowledge, nothing else has changed – it had been running for well over a year without any issues.



 Comments   
Comment by Kelsey Schubert [ 01/Sep/17 ]

I'm glad you were able to correct the issue.

Kind regards,
Thomas

Comment by Dennis Andersen [ 01/Sep/17 ]

This ticket can be closed. It turns out that the user name that mongod was running under was changed, causing permissions issues.

Thanks.

Generated at Thu Feb 08 04:25:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.