[SERVER-17613] Unable to start mongod after unclean shutdown Created: 16/Mar/15  Updated: 13/May/15  Resolved: 27/Mar/15

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.0.1
Fix Version/s: 3.0.2, 3.1.1

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Susan LoVerso
Resolution: Done Votes: 0
Labels: 28qa, wiredtiger
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File wt301-invalidaccess.tar.gz    
Issue Links:
Depends
is depended on by WT-1901 Issues resolved in WiredTiger 2.5.3 Closed
Duplicate
is duplicated by SERVER-17824 Mongo server seagfault Closed
is duplicated by SERVER-17569 Got signal: 11 (Segmentation fault). Closed
Related
is related to SERVER-17152 WiredTiger file corrupted during powe... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   
Issue Status as of Apr 02, 2015

ISSUE SUMMARY
After an ungraceful termination, a mongod running WiredTiger may fail to read the journal and fail to start, showing the following messages in the logs:

2015-03-16T14:49:24.992-0400 W -        [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty.
2015-03-16T14:49:24.992-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
...
2015-03-16T14:49:25.075-0400 F -        [initandlisten] Invalid access at address: 0

WORKAROUNDS
Please upgrade to to 3.0.2.

Alternatively, a mongod from 3.0.0-rc8 can be used to recover the data files, but this is not recommended.

AFFECTED VERSIONS
3.0.0-rc8+ and 3.0.1

FIX VERSION
The fix is included in the 3.0.2 production release.

Original description

As part of the Power cycle testing (see SERVER-17152), the following was discovered:

jonathan@CAP-JA-Ubuntu14:~$ sudo mongod --dbpath /data/wiredTiger --storageEngine wiredTiger
2015-03-16T14:49:24.992-0400 W -        [initandlisten] Detected unclean shutdown - /data/wiredTiger/mongod.lock is not empty.
2015-03-16T14:49:24.992-0400 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2015-03-16T14:49:24.992-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-03-16T14:49:25.075-0400 F -        [initandlisten] Invalid access at address: 0
2015-03-16T14:49:25.083-0400 F -        [initandlisten] Got signal: 11 (Segmentation fault).
 
 0xf4fd39 0xf4f3b2 0xf4f70e 0x7f4bac390340 0x13576e8 0x132ad0d 0x132b31b 0x133a2ca 0x138c8a5 0x131b3c1 0x13150dc 0xd62cab 0xd60b28 0xa6f8bd 0x7e20c0 0x7e7704 0x7f4bab3b0ec5 0x7e02c9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B4FD39"},{"b":"400000","o":"B4F3B2"},{"b":"400000","o":"B4F70E"},{"b":"7F4BAC380000","o":"10340"},{"b":"400000","o":"F576E8"},{"b":"400000","o":"F2AD0D"},{"b":"400000","o":"F2B31B"},{"b":"400000","o":"F3A2CA"},{"b":"400000","o":"F8C8A5"},{"b":"400000","o":"F1B3C1"},{"b":"400000","o":"F150DC"},{"b":"400000","o":"962CAB"},{"b":"400000","o":"960B28"},{"b":"400000","o":"66F8BD"},{"b":"400000","o":"3E20C0"},{"b":"400000","o":"3E7704"},{"b":"7F4BAB38F000","o":"21EC5"},{"b":"400000","o":"3E02C9"}],"processInfo":{ "mongodbVersion" : "3.0.1-rc0", "gitVersion" : "3f999d764cdd307f779b27e3a28230b21bd43e25", "uname" : { "sysname" : "Linux", "release" : "3.16.0-31-generic", "version" : "#41~14.04.1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF64135000", "elfType" : 3 }, { "b" : "7F4BAC380000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F4BAC178000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F4BABF74000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F4BABC70000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F4BAB96A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F4BAB754000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F4BAB38F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F4BAC59E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4fd39]
 mongod(+0xB4F3B2) [0xf4f3b2]
 mongod(+0xB4F70E) [0xf4f70e]
 libpthread.so.0(+0x10340) [0x7f4bac390340]
 mongod(__wt_struct_unpack+0x368) [0x13576e8]
 mongod(+0xF2AD0D) [0x132ad0d]
 mongod(+0xF2B31B) [0x132b31b]
 mongod(__wt_log_needs_recovery+0xCA) [0x133a2ca]
 mongod(__wt_txn_recover+0x425) [0x138c8a5]
 mongod(__wt_connection_workers+0x61) [0x131b3c1]
 mongod(wiredtiger_open+0x118C) [0x13150dc]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd62cab]
 mongod(+0x960B28) [0xd60b28]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa6f8bd]
 mongod(_ZN5mongo13initAndListenEi+0x2F0) [0x7e20c0]
 mongod(main+0x134) [0x7e7704]
 libc.so.6(__libc_start_main+0xF5) [0x7f4bab3b0ec5]
 mongod(+0x3E02C9) [0x7e02c9]
-----  END BACKTRACE  -----



 Comments   
Comment by Michael Cahill (Inactive) [ 27/Mar/15 ]

Resolved with latest drop from WT.

Generated at Thu Feb 08 03:45:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.