[SERVER-32917] mongodb 3.4.10 crash Created: 26/Jan/18  Updated: 22/Feb/18  Resolved: 30/Jan/18

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.4.10
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: balbo liu Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongod.conf     File mongod.log.1     File mongodb.conf     Text File mongodb.log    
Participants:

 Description   

mongodb 3.4.10 crash with the following log message:

2018-01-26T08:55:58.148+0800 F JOURNAL  [journal writer] exception in journalWriterThread causing immediate shutdown: boost::filesystem::current_path: No such file or directory
2018-01-26T08:55:58.148+0800 I -        [journal writer] Invariant failure false src/mongo/db/storage/mmap_v1/dur_journal_writer.cpp 272
2018-01-26T08:55:58.148+0800 I -        [journal writer]
 
***aborting after invariant() failure
 
 
2018-01-26T08:55:58.186+0800 F -        [journal writer] Got signal: 6 (Aborted).
 
 0x7f40a6c6a581 0x7f40a6c69799 0x7f40a6c69c7d 0x7f40a435c330 0x7f40a3fbdc37 0x7f40a3fc1028 0x7f40a5f0de78 0x7f40a68e49d9 0x7f40a76e1bb0 0x7f40a4354184 0x7f40a408137d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F40A56F2000","o":"1578581","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F40A56F2000","o":"1577799"},{"b":"7F40A56F2000","o":"1577C7D"},{"b":"7F40A434C000","o":"10330"},{"b":"7F40A3F87000","o":"36C37","s":"gsignal"},{"b":"7F40A3F87000","o":"3A028","s":"abort"},{"b":"7F40A56F2000","o":"81BE78","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"7F40A56F2000","o":"11F29D9","s":"_ZN5mongo3dur13JournalWriter20_journalWriterThreadEv"},{"b":"7F40A56F2000","o":"1FEFBB0"},{"b":"7F40A434C000","o":"8184"},{"b":"7F40A3F87000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-86-generic", "version" : "#131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "7F40A56F2000", "elfType" : 3, "buildId" : "D21A02CF9C7297563D25465FBECF02F5CA47FBAD" }, { "b" : "7FFEB3880000", "elfType" : 3, "buildId" : "FB51781B01C799D4B6F6FED07F9862B3B24B375F" }, { "b" : "7F40A526E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F40A4E92000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "607C4A6F3428B6F065D55691AFDC4E343786A8A5" }, { "b" : "7F40A4C8A000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F40A4A86000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F40A4780000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F40A456A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F40A434C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F40A3F87000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F40A54CD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f40a6c6a581]
 mongod(+0x1577799) [0x7f40a6c69799]
 mongod(+0x1577C7D) [0x7f40a6c69c7d]
 libpthread.so.0(+0x10330) [0x7f40a435c330]
 libc.so.6(gsignal+0x37) [0x7f40a3fbdc37]
 libc.so.6(abort+0x148) [0x7f40a3fc1028]
 mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0x0) [0x7f40a5f0de78]
 mongod(_ZN5mongo3dur13JournalWriter20_journalWriterThreadEv+0x1009) [0x7f40a68e49d9]
 mongod(+0x1FEFBB0) [0x7f40a76e1bb0]
 libpthread.so.0(+0x8184) [0x7f40a4354184]
 libc.so.6(clone+0x6D) [0x7f40a408137d]
-----  END BACKTRACE  -----



 Comments   
Comment by Mark Agarunov [ 30/Jan/18 ]

Hello balbo,

Thank you for providing this information. Looking at the the output you've provided, it appears that some of the files in your $dbpath are owned by root instead of the mongod user. Please verify that the permissions are correct for the user you are running mongod with as the error you are seeing generally indicates a permissions or access issue on the underlying filesystem.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thanks,
Mark

Comment by balbo liu [ 27/Jan/18 ]

these are some commands output:

df -h
Filesystem  Mounted on
udev        /dev
tmpfs       /run
/dev/xvda1  /
none        /sys/fs/cgroup
none        /run/lock
none        /run/shm
none        /run/user
 
 
ls -l /var/lib/mongodb/journal/
-rw------- 1 root    root    1073741824 Jan 26 10:34 j._0
-rw------- 1 root    root            88 Jan 26 10:34 lsn
-rw------- 1 mongodb mongodb 1073741824 Dec 11  2016 prealloc.1
-rw------- 1 mongodb mongodb 1073741824 Dec 11  2016 prealloc.2
 
 
/ls -l /var/lib/mongodb
drwxr-xr-x 2 mongodb mongodb     4096 Jan  6 21:19 admin
-rw------- 1 root    root    67108864 Jan  6 21:44 admin.0
-rw------- 1 root    root    16777216 Jan  6 21:41 admin.ns
drwxr-xr-x 2 root    root        4096 Jan 27 10:59 diagnostic.data
drwxr-xr-x 2 mongodb mongodb     4096 Jan  6 21:19 forum
-rw------- 1 root    root    67108864 Jan 26 10:34 forum.0
-rw------- 1 root    root    16777216 Jan 26 10:34 forum.ns
drwxr-xr-x 2 mongodb mongodb     4096 Jan 26 10:34 journal
drwxr-xr-x 2 mongodb mongodb     4096 Jan  6 21:19 local
-rw------- 1 root    root    67108864 Jan 26 10:23 local.0
-rw------- 1 root    root    16777216 Jan 26 10:23 local.ns
-rw-r--r-- 1 mongodb mongodb        6 Jan 26 10:23 mongod.lock
-rw-r--r-- 1 mongodb mongodb       69 Jan  6 20:55 storage.bson

Comment by balbo liu [ 27/Jan/18 ]

"Has the dbpath been manually modified outside of mongodb in any way?" No

the mongod.log file is empty, I upload the mongod.log.1

"the startup of the node" what is it ?

Comment by Mark Agarunov [ 26/Jan/18 ]

Hello balbo,

Thank you for the report. Looking over this, I'm seeing a file not found error:

2018-01-26T08:55:58.148+0800 F JOURNAL  [journal writer] exception in journalWriterThread causing immediate shutdown: boost::filesystem::current_path: No such file or directory

Has the dbpath been manually modified outside of mongodb in any way? To get a better idea of why you may be seeing this error, could you please provide the complete log files from mongod, including the startup of the node.

Thanks
Mark

Generated at Thu Feb 08 04:31:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.