[SERVER-18850] Cannot start or --repair after WiredTiger.wt corruption (due to unclean shutdown) Created: 05/Jun/15  Updated: 03/Jul/15  Resolved: 02/Jul/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Hey Pas Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.turtle.new     File WiredTiger.wt     File WiredTiger.wt.new    
Issue Links:
Duplicate
duplicates SERVER-18316 Database with WT engine fails to reco... Closed
Operating System: ALL
Steps To Reproduce:

an unlucky unclear shutdown, probably.

Participants:

 Description   

Hello!

Unfortunately we've encountered a nasty metadata corruption, which prevents us from recovering this - luckly just a small, but kind of test-in-prod - database.

If I can assist in debugging this, please don't hesitate to contact me.

mongod.log.2015-06-04T15-29-41 (this file seems funky when viewed with less, though hexdump -c only shows a few \0 bytes at the end, but not as much as less):

2015-06-01T18:06:47.659+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-01T18:07:00.407+0200 I CONTROL  [initandlisten] MongoDB starting : pid=887 port=27017 dbpath=/var/lib/mongodb 64-bit host=redacted
2015-06-01T18:07:00.407+0200 I CONTROL  [initandlisten] db version v3.0.3
2015-06-01T18:07:00.407+0200 I CONTROL  [initandlisten] git version: b40106b36eecd1b4407eb1ad1af6bc60593c6105
2015-06-01T18:07:00.407+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2015-06-01T18:07:00.408+0200 I CONTROL  [initandlisten] build info: Linux ip-10-225-179-153 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-06-01T18:07:00.408+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2015-06-01T18:07:00.408+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", processManagement: { fork: false }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger" }, systemLog: { destination: "file", path: "/var/log/mongodb/mongod.log", quiet: true } }
2015-06-01T18:07:00.500+0200 I NETWORK  [initandlisten] waiting for connections on port 27017
2015-06-01T18:07:03.717+0200 I WRITE    [conn2] insert test_shit.testtest query: { _id: ObjectId('556c8327e5e61e0391fb1ce9'), aaa: "asdasd" } ninserted:1 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { W: 1 } } } 113ms
2015-06-01T18:07:03.717+0200 I COMMAND  [conn2] command test_shit.$cmd command: insert { insert: "testtest", ordered: true, documents: [ { _id: ObjectId('556c8327e5e61e0391fb1ce9'), aaa: "asdasd" } ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:40 locks:{ Global: { acquireCount: { w: 2 } }, Database: { acquireCount: { w: 1, W: 1 } }, Collection: { acquireCount: { W: 1 } } } 138ms
2015-06-01T18:07:03.717+0200 I COMMAND  [conn2] dropDatabase test_shit starting
2015-06-01T18:07:03.718+0200 I COMMAND  [conn2] dropDatabase test_shit finished
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
mongod.log.2015-06-04T15-29-41 (END)

mongod.log.2015-06-04T15-53-40:

2015-06-04T17:29:41.956+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-04T17:29:41.964+0200 E STORAGE  [initandlisten] WiredTiger (0) [1433431781:964100][841:0x7fe62f4bcb80], file:WiredTiger.wt, connection: read checksum error [4096B @ 24576, 1890799341 != 1416930015]
2015-06-04T17:29:41.964+0200 E STORAGE  [initandlisten] WiredTiger (0) [1433431781:964152][841:0x7fe62f4bcb80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-04T17:29:41.964+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1433431781:964164][841:0x7fe62f4bcb80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-04T17:29:41.964+0200 I -        [initandlisten] Fatal Assertion 28558
2015-06-04T17:29:42.214+0200 I CONTROL  [initandlisten] 
 0xf51949 0xef1671 0xed6261 0xd7b2ba 0x13816c9 0x1381885 0x1381d24 0x12d578e 0x12d5c28 0x12d2fe3 0x12d6956 0x12ef431 0x1317a0b 0x1380b43 0x134eafb 0x13150f7 0xd65c2b 0xd63c28 0xa81dbd 0x808712 0x7d5334 0x7fe62da64a40 0x8064d7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B51949"},{"b":"400000","o":"AF1671"},{"b":"400000","o":"AD6261"},{"b":"400000","o":"97B2BA"},{"b":"400000","o":"F816C9"},{"b":"400000","o":"F81885"},{"b":"400000","o":"F81D24"},{"b":"400000","o":"ED578E"},{"b":"400000","o":"ED5C28"},{"b":"400000","o":"ED2FE3"},{"b":"400000","o":"ED6956"},{"b":"400000","o":"EEF431"},{"b":"400000","o":"F17A0B"},{"b":"400000","o":"F80B43"},{"b":"400000","o":"F4EAFB"},{"b":"400000","o":"F150F7"},{"b":"400000","o":"965C2B"},{"b":"400000","o":"963C28"},{"b":"400000","o":"681DBD"},{"b":"400000","o":"408712"},{"b":"400000","o":"3D5334"},{"b":"7FE62DA44000","o":"20A40"},{"b":"400000","o":"4064D7"}],"processInfo":{ "mongodbVersion" : "3.0.3", "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105", "uname" : { "sysname" : "Linux", "release" : "3.19.0-18-generic", "version" : "#18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "F56F80CB96B4DBFC070BEB0ADAC7D6B274BFC6B1" }, { "b" : "7FFC17F5B000", "elfType" : 3, "buildId" : "F9FFD7E8DF98D057DEE553A5CB80D669FD0969CE" }, { "b" : "7FE62F08A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "A37A144BCBEE86A9E02DFF5021A111EDE6A1F212" }, { "b" : "7FE62EE2B000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "4F45CC86BB893312DFF5A9E01C7BF0744936E845" }, { "b" : "7FE62EA47000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "323B71606BA892F0EF8C7923FF2806D2E382309F" }, { "b" : "7FE62E83F000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CB26CB6169CBAA9DA2E38A70BCF7D57A8047CCF3" }, { "b" : "7FE62E63B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "F974B99C0C327670EF882BA13912E995A12C6402" }, { "b" : "7FE62E32C000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4A9F593569BBBD9D4AAC7027DE09E016A6F3E1DB" }, { "b" : "7FE62E024000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "372ADD7EB11B9D51E634BC91DAAA58DF90567FD1" }, { "b" : "7FE62DE0E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "640D535660B2B8EBB1611AC2185F2385557C9D18" }, { "b" : "7FE62DA44000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8ACD43CF74A9756CD727B8516B08679EE071A92D" }, { "b" : "7FE62F2A8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "903BB7A6DEEFD966DCEEC4566C70444C727ED294" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf51949]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xef1671]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xed6261]
 mongod(+0x97B2BA) [0xd7b2ba]
 mongod(__wt_eventv+0x489) [0x13816c9]
 mongod(__wt_err+0x95) [0x1381885]
 mongod(__wt_panic+0x24) [0x1381d24]
 mongod(__wt_block_extlist_read+0x6E) [0x12d578e]
 mongod(__wt_block_extlist_read_avail+0x28) [0x12d5c28]
 mongod(__wt_block_checkpoint_load+0x193) [0x12d2fe3]
 mongod(+0xED6956) [0x12d6956]
 mongod(__wt_btree_open+0xAB1) [0x12ef431]
 mongod(__wt_conn_btree_get+0x19B) [0x1317a0b]
 mongod(__wt_session_get_btree+0x343) [0x1380b43]
 mongod(__wt_metadata_open+0x2B) [0x134eafb]
 mongod(wiredtiger_open+0xCD7) [0x13150f7]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2EB) [0xd65c2b]
 mongod(+0x963C28) [0xd63c28]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa81dbd]
 mongod(_ZN5mongo13initAndListenEi+0x422) [0x808712]
 mongod(main+0x134) [0x7d5334]
 libc.so.6(__libc_start_main+0xF0) [0x7fe62da64a40]
 mongod(+0x4064D7) [0x8064d7]
-----  END BACKTRACE  -----
2015-06-04T17:29:42.214+0200 I -        [initandlisten] 
 
***aborting after fassert() failure

/etc/mongod.conf:

storage:
    dbPath: "/var/lib/mongodb"
    engine: "wiredTiger"
systemLog:
    destination: file
    path: "/var/log/mongodb/mongod.log"
processManagement:
    fork: false

trying --repair with

db version v3.1.4-pre-
git version: 814125d3b17017224c2e17c8cee778184b8af7a6
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014

./mongod --repair --dbpath /var/lib/mongodb

2015-06-06T00:10:10.378+0200 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2015-06-06T00:10:10.378+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-06T00:10:10.434+0200 E STORAGE  [initandlisten] WiredTiger (0) [1433542210:434822][1624:0x7fdc82932b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 24576, 1890799341 != 1416930015]
2015-06-06T00:10:10.435+0200 E STORAGE  [initandlisten] WiredTiger (0) [1433542210:435016][1624:0x7fdc82932b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-06T00:10:10.435+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1433542210:435052][1624:0x7fdc82932b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-06T00:10:10.435+0200 I -        [initandlisten] Fatal Assertion 28558
2015-06-06T00:10:10.482+0200 I CONTROL  [initandlisten] 
 0xfe63b6 0xf91718 0xf7953d 0xdf9184 0x13ea409 0x13ea5b5 0x13ea9c4 0x134605e 0x13464f8 0x1344103 0x1347226 0x135efb1 0x13867a7 0x13e93d6 0x13e97b2 0x13ba1db 0x13842ee 0xde619e 0xde3dfb 0xd21266 0x85744a 0x818ce4 0x7fdc80ed9a40 0x855809
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BE63B6"},{"b":"400000","o":"B91718"},{"b":"400000","o":"B7953D"},{"b":"400000","o":"9F9184"},{"b":"400000","o":"FEA409"},{"b":"400000","o":"FEA5B5"},{"b":"400000","o":"FEA9C4"},{"b":"400000","o":"F4605E"},{"b":"400000","o":"F464F8"},{"b":"400000","o":"F44103"},{"b":"400000","o":"F47226"},{"b":"400000","o":"F5EFB1"},{"b":"400000","o":"F867A7"},{"b":"400000","o":"FE93D6"},{"b":"400000","o":"FE97B2"},{"b":"400000","o":"FBA1DB"},{"b":"400000","o":"F842EE"},{"b":"400000","o":"9E619E"},{"b":"400000","o":"9E3DFB"},{"b":"400000","o":"921266"},{"b":"400000","o":"45744A"},{"b":"400000","o":"418CE4"},{"b":"7FDC80EB9000","o":"20A40"},{"b":"400000","o":"455809"}],"processInfo":{ "mongodbVersion" : "3.1.4-pre-", "gitVersion" : "814125d3b17017224c2e17c8cee778184b8af7a6", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-18-generic", "version" : "#18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "2B704624C59DB68E699CEBFACBC71D060E0748F4" }, { "b" : "7FFC5D868000", "elfType" : 3, "buildId" : "F9FFD7E8DF98D057DEE553A5CB80D669FD0969CE" }, { "b" : "7FDC824BE000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "8196D46E3065596B50170E44427C2C73A3FE25FD" }, { "b" : "7FDC820DA000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D680AC0C233C0C936A4942528431C328A6F6EE46" }, { "b" : "7FDC81ED2000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CB26CB6169CBAA9DA2E38A70BCF7D57A8047CCF3" }, { "b" : "7FDC81CCE000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "F974B99C0C327670EF882BA13912E995A12C6402" }, { "b" : "7FDC819BF000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4A9F593569BBBD9D4AAC7027DE09E016A6F3E1DB" }, { "b" : "7FDC816B7000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "372ADD7EB11B9D51E634BC91DAAA58DF90567FD1" }, { "b" : "7FDC814A1000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "640D535660B2B8EBB1611AC2185F2385557C9D18" }, { "b" : "7FDC81283000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "A37A144BCBEE86A9E02DFF5021A111EDE6A1F212" }, { "b" : "7FDC80EB9000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8ACD43CF74A9756CD727B8516B08679EE071A92D" }, { "b" : "7FDC8271E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "903BB7A6DEEFD966DCEEC4566C70444C727ED294" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xfe63b6]
 mongod(_ZN5mongo10logContextEPKc+0x138) [0xf91718]
 mongod(_ZN5mongo13fassertFailedEi+0x4D) [0xf7953d]
 mongod(+0x9F9184) [0xdf9184]
 mongod(__wt_eventv+0x3E9) [0x13ea409]
 mongod(__wt_err+0x95) [0x13ea5b5]
 mongod(__wt_panic+0x24) [0x13ea9c4]
 mongod(__wt_block_extlist_read+0x6E) [0x134605e]
 mongod(__wt_block_extlist_read_avail+0x28) [0x13464f8]
 mongod(__wt_block_checkpoint_load+0x193) [0x1344103]
 mongod(+0xF47226) [0x1347226]
 mongod(__wt_btree_open+0xB21) [0x135efb1]
 mongod(__wt_conn_btree_open+0x147) [0x13867a7]
 mongod(__wt_session_get_btree+0x176) [0x13e93d6]
 mongod(__wt_session_get_btree+0x552) [0x13e97b2]
 mongod(__wt_metadata_open+0x2B) [0x13ba1db]
 mongod(wiredtiger_open+0x128E) [0x13842ee]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2FE) [0xde619e]
 mongod(+0x9E3DFB) [0xde3dfb]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x5E6) [0xd21266]
 mongod(_ZN5mongo13initAndListenEi+0x16A) [0x85744a]
 mongod(main+0x134) [0x818ce4]
 libc.so.6(__libc_start_main+0xF0) [0x7fdc80ed9a40]
 mongod(+0x455809) [0x855809]
-----  END BACKTRACE  -----
2015-06-06T00:10:10.482+0200 I -        [initandlisten] 
 
***aborting after fassert() failure

The whole datadir is just just a few megabytes.

Thanks!

Also, it seems someone else has this problem too: http://serverfault.com/questions/696972/corrupt-wiredtiger-mongodb-files



 Comments   
Comment by Ramon Fernandez Marina [ 03/Jul/15 ]

pasthelod@gmail.com, I forgot to add that, after recovering from this situation, it's advisable to run repair validate to make sure there are no other inconsistencies in your data.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 02/Jul/15 ]

Thanks for the update pasthelod@gmail.com. Please note that 3.1.5-pre was replaced by 3.1.5, but also note that this is a development release not recommended for production. My recommendation would be to use 3.0.4 if you intend to run a production system.

It would be preferable to keep the attachments for historical purposes if that's fine with you. Glad to hear that your installation is back up and running, I'm going to close this ticket as a duplicate of SERVER-18316.

Regards,
Ramón.

Comment by Hey Pas [ 02/Jul/15 ]

I can confirm that 3.1.5-pre (546e2943af3c9d0eb50062c26f52303452423f29) can open the database and everything seems to be working fine.

Can I assume that it's okay to delete the attachments?

Comment by Ramon Fernandez Marina [ 02/Jul/15 ]

pasthelod@gmail.com, I've uploaded these two files after the repair attempt with the .new suffix. Can you please put them in your dbpath without the suffix and with the right permissions, and try to start your mongod instance??

Thanks,
Ramón.

Comment by Ramon Fernandez Marina [ 02/Jul/15 ]

pasthelod@gmail.com, please upload the WiredTiger.wt and WiredTiger.turtle and we'll take a look.

Comment by Hey Pas [ 02/Jul/15 ]

Hello,

Sorry for missing your previous mail/comment!

Yes, I'm interested in some kind of recovery. Though, can I mangle the DB and collection names in the .wt file somehow? It's not secret, and the data is not that confidential, but if it can be done with relative ease (so they are not checksummed), I'll just sed the file and upload them.

Thanks!

Comment by Ramon Fernandez Marina [ 01/Jul/15 ]

pasthelod@gmail.com, I'm going to close this ticket for now since I haven't heard from you in a while. If you're still interested in attempting to recover this instance please upload the WiredTiger.wt and WiredTiger.turtle files and we'll reopen this ticket for further investigation.

Regards,
Ramón.

Comment by Ramon Fernandez Marina [ 19/Jun/15 ]

pasthelod@gmail.com, apologies if I wasn't clear: 3.0.4 prevents the problem from appearing, but if the problem is present it won't fix it. Can you please zip up your WiredTiger.wt and WiredTiger.turtle so we can further investigate?

Thanks,
Ramón.

Comment by Hey Pas [ 15/Jun/15 ]

Sadly, I get the same result.

Here's 3.0.4-rc0

 bin/mongod --repair --dbpath /var/lib/mongodb --storageEngine wiredTiger
2015-06-15T18:38:55.187+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-15T18:38:55.196+0200 E STORAGE  [initandlisten] WiredTiger (0) [1434386335:196939][8579:0x7fe7ca6dfb80], file:WiredTiger.wt, connection: read checksum error [4096B @ 24576, 1890799341 != 1416930015]
2015-06-15T18:38:55.197+0200 E STORAGE  [initandlisten] WiredTiger (0) [1434386335:197020][8579:0x7fe7ca6dfb80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-15T18:38:55.197+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1434386335:197047][8579:0x7fe7ca6dfb80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-15T18:38:55.197+0200 I -        [initandlisten] Fatal Assertion 28558
2015-06-15T18:38:55.215+0200 I CONTROL  [initandlisten] 
 0xf7a0f9 0xf19121 0xefdbb1 0xda3d5a 0x13ab769 0x13ab925 0x13abdc4 0x12fe17e 0x12fe618 0x12fb9a3 0x12ff346 0x1318601 0x134198b 0x13aabe0 0x1378b1b 0x133f077 0xd8e6f0 0xd8c648 0xa9baed 0x825fb2 0x7f2829 0x7fe7c8c86a40 0x823da9
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B7A0F9"},{"b":"400000","o":"B19121"},{"b":"400000","o":"AFDBB1"},{"b":"400000","o":"9A3D5A"},{"b":"400000","o":"FAB769"},{"b":"400000","o":"FAB925"},{"b":"400000","o":"FABDC4"},{"b":"400000","o":"EFE17E"},{"b":"400000","o":"EFE618"},{"b":"400000","o":"EFB9A3"},{"b":"400000","o":"EFF346"},{"b":"400000","o":"F18601"},{"b":"400000","o":"F4198B"},{"b":"400000","o":"FAABE0"},{"b":"400000","o":"F78B1B"},{"b":"400000","o":"F3F077"},{"b":"400000","o":"98E6F0"},{"b":"400000","o":"98C648"},{"b":"400000","o":"69BAED"},{"b":"400000","o":"425FB2"},{"b":"400000","o":"3F2829"},{"b":"7FE7C8C66000","o":"20A40"},{"b":"400000","o":"423DA9"}],"processInfo":{ "mongodbVersion" : "3.0.4-rc0", "gitVersion" : "e4901b00e9de692b404e9a271fdee05723c52f00", "uname" : { "sysname" : "Linux", "release" : "3.19.0-18-generic", "version" : "#18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "D50EB252A841E87D59B76CBCDF31C0CCE29D87B6" }, { "b" : "7FFCFFB32000", "elfType" : 3, "buildId" : "F9FFD7E8DF98D057DEE553A5CB80D669FD0969CE" }, { "b" : "7FE7CA2AD000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "A37A144BCBEE86A9E02DFF5021A111EDE6A1F212" }, { "b" : "7FE7CA04D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "8196D46E3065596B50170E44427C2C73A3FE25FD" }, { "b" : "7FE7C9C69000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D680AC0C233C0C936A4942528431C328A6F6EE46" }, { "b" : "7FE7C9A61000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CB26CB6169CBAA9DA2E38A70BCF7D57A8047CCF3" }, { "b" : "7FE7C985D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "F974B99C0C327670EF882BA13912E995A12C6402" }, { "b" : "7FE7C954E000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4A9F593569BBBD9D4AAC7027DE09E016A6F3E1DB" }, { "b" : "7FE7C9246000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "372ADD7EB11B9D51E634BC91DAAA58DF90567FD1" }, { "b" : "7FE7C9030000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "640D535660B2B8EBB1611AC2185F2385557C9D18" }, { "b" : "7FE7C8C66000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8ACD43CF74A9756CD727B8516B08679EE071A92D" }, { "b" : "7FE7CA4CB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "903BB7A6DEEFD966DCEEC4566C70444C727ED294" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf7a0f9]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf19121]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xefdbb1]
 mongod(+0x9A3D5A) [0xda3d5a]
 mongod(__wt_eventv+0x489) [0x13ab769]
 mongod(__wt_err+0x95) [0x13ab925]
 mongod(__wt_panic+0x24) [0x13abdc4]
 mongod(__wt_block_extlist_read+0x6E) [0x12fe17e]
 mongod(__wt_block_extlist_read_avail+0x28) [0x12fe618]
 mongod(__wt_block_checkpoint_load+0x193) [0x12fb9a3]
 mongod(+0xEFF346) [0x12ff346]
 mongod(__wt_btree_open+0xAB1) [0x1318601]
 mongod(__wt_conn_btree_get+0x19B) [0x134198b]
 mongod(__wt_session_get_btree+0x500) [0x13aabe0]
 mongod(__wt_metadata_open+0x2B) [0x1378b1b]
 mongod(wiredtiger_open+0xCD7) [0x133f077]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x300) [0xd8e6f0]
 mongod(+0x98C648) [0xd8c648]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x30D) [0xa9baed]
 mongod(_ZN5mongo13initAndListenEi+0x422) [0x825fb2]
 mongod(main+0x139) [0x7f2829]
 libc.so.6(__libc_start_main+0xF0) [0x7fe7c8c86a40]
 mongod(+0x423DA9) [0x823da9]
-----  END BACKTRACE  -----
2015-06-15T18:38:55.215+0200 I -        [initandlisten] 
 
***aborting after fassert() failure

Latest nightly (debian 7.1)

 bin/mongod --repair --dbpath /var/lib/mongodb --storageEngine wiredTiger
2015-06-15T18:39:46.589+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-06-15T18:39:46.599+0200 E STORAGE  [initandlisten] WiredTiger (0) [1434386386:599728][8611:0x7fe880ea2b80], file:WiredTiger.wt, connection: read checksum error [4096B @ 24576, 1890799341 != 1416930015]
2015-06-15T18:39:46.599+0200 E STORAGE  [initandlisten] WiredTiger (0) [1434386386:599824][8611:0x7fe880ea2b80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2015-06-15T18:39:46.599+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1434386386:599851][8611:0x7fe880ea2b80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2015-06-15T18:39:46.599+0200 I -        [initandlisten] Fatal Assertion 28558
2015-06-15T18:39:46.619+0200 I CONTROL  [initandlisten] 
 0xfe63b6 0xf91718 0xf7953d 0xdf9184 0x13ea409 0x13ea5b5 0x13ea9c4 0x134605e 0x13464f8 0x1344103 0x1347226 0x135efb1 0x13867a7 0x13e93d6 0x13e97b2 0x13ba1db 0x13842ee 0xde619e 0xde3dfb 0xd21266 0x85744a 0x818ce4 0x7fe87f449a40 0x855809
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BE63B6"},{"b":"400000","o":"B91718"},{"b":"400000","o":"B7953D"},{"b":"400000","o":"9F9184"},{"b":"400000","o":"FEA409"},{"b":"400000","o":"FEA5B5"},{"b":"400000","o":"FEA9C4"},{"b":"400000","o":"F4605E"},{"b":"400000","o":"F464F8"},{"b":"400000","o":"F44103"},{"b":"400000","o":"F47226"},{"b":"400000","o":"F5EFB1"},{"b":"400000","o":"F867A7"},{"b":"400000","o":"FE93D6"},{"b":"400000","o":"FE97B2"},{"b":"400000","o":"FBA1DB"},{"b":"400000","o":"F842EE"},{"b":"400000","o":"9E619E"},{"b":"400000","o":"9E3DFB"},{"b":"400000","o":"921266"},{"b":"400000","o":"45744A"},{"b":"400000","o":"418CE4"},{"b":"7FE87F429000","o":"20A40"},{"b":"400000","o":"455809"}],"processInfo":{ "mongodbVersion" : "3.1.4-pre-", "gitVersion" : "814125d3b17017224c2e17c8cee778184b8af7a6", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.19.0-18-generic", "version" : "#18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "2B704624C59DB68E699CEBFACBC71D060E0748F4" }, { "b" : "7FFDE35A5000", "elfType" : 3, "buildId" : "F9FFD7E8DF98D057DEE553A5CB80D669FD0969CE" }, { "b" : "7FE880A2E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "8196D46E3065596B50170E44427C2C73A3FE25FD" }, { "b" : "7FE88064A000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "D680AC0C233C0C936A4942528431C328A6F6EE46" }, { "b" : "7FE880442000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CB26CB6169CBAA9DA2E38A70BCF7D57A8047CCF3" }, { "b" : "7FE88023E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "F974B99C0C327670EF882BA13912E995A12C6402" }, { "b" : "7FE87FF2F000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4A9F593569BBBD9D4AAC7027DE09E016A6F3E1DB" }, { "b" : "7FE87FC27000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "372ADD7EB11B9D51E634BC91DAAA58DF90567FD1" }, { "b" : "7FE87FA11000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "640D535660B2B8EBB1611AC2185F2385557C9D18" }, { "b" : "7FE87F7F3000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "A37A144BCBEE86A9E02DFF5021A111EDE6A1F212" }, { "b" : "7FE87F429000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8ACD43CF74A9756CD727B8516B08679EE071A92D" }, { "b" : "7FE880C8E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "903BB7A6DEEFD966DCEEC4566C70444C727ED294" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xfe63b6]
 mongod(_ZN5mongo10logContextEPKc+0x138) [0xf91718]
 mongod(_ZN5mongo13fassertFailedEi+0x4D) [0xf7953d]
 mongod(+0x9F9184) [0xdf9184]
 mongod(__wt_eventv+0x3E9) [0x13ea409]
 mongod(__wt_err+0x95) [0x13ea5b5]
 mongod(__wt_panic+0x24) [0x13ea9c4]
 mongod(__wt_block_extlist_read+0x6E) [0x134605e]
 mongod(__wt_block_extlist_read_avail+0x28) [0x13464f8]
 mongod(__wt_block_checkpoint_load+0x193) [0x1344103]
 mongod(+0xF47226) [0x1347226]
 mongod(__wt_btree_open+0xB21) [0x135efb1]
 mongod(__wt_conn_btree_open+0x147) [0x13867a7]
 mongod(__wt_session_get_btree+0x176) [0x13e93d6]
 mongod(__wt_session_get_btree+0x552) [0x13e97b2]
 mongod(__wt_metadata_open+0x2B) [0x13ba1db]
 mongod(wiredtiger_open+0x128E) [0x13842ee]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x2FE) [0xde619e]
 mongod(+0x9E3DFB) [0xde3dfb]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x5E6) [0xd21266]
 mongod(_ZN5mongo13initAndListenEi+0x16A) [0x85744a]
 mongod(main+0x134) [0x818ce4]
 libc.so.6(__libc_start_main+0xF0) [0x7fe87f449a40]
 mongod(+0x455809) [0x855809]
-----  END BACKTRACE  -----
2015-06-15T18:39:46.619+0200 I -        [initandlisten] 
 
***aborting after fassert() failure

Comment by Ramon Fernandez Marina [ 11/Jun/15 ]

Thanks for your report pasthelod@gmail.com. You're probably running into SERVER-18316, which is the ticket where we've consolidated all the issues related to unclean shutdowns triggering file corruption.

This issue is fixed in MongoDB 3.0.4, scheduled for next week – but we've already release the 3.0.4-rc0 release candidate so it would be very useful if you could test it on your end and confirms that it does indeed solve the issue.

Can you please download 3.0.4-rc0 to test it on your end and report back with the results?

Thanks,
Ramón.

Generated at Thu Feb 08 03:48:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.