[SERVER-34295] WiredTiger.wt Read checksum error after accidentally reboot of Server Created: 04/Apr/18  Updated: 27/Jul/18  Resolved: 10/Apr/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.12
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Martins Lui Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: envm, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.lock     File WiredTiger.turtle     File WiredTiger.wt     Text File mongod.log     File repair_attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi,

I have accidentally reboot the mongodb server (AWS EC2) and the mongod now won't be able to boot. I have Googled for similar issues and end up here. I have tried using `--repair` on the ./data directory but no luck. Seems to me the only way is to seek help from the development team here and fix the checksum of the WiredTiger.wt. I have attached the related files and please kindly help to repair the file and I will be forever grateful.

2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1713 port=27017 dbpath=/home/datafeed/luxify-scrap/./data 64-bit host=ip-172-31-24-153
2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] db version v3.0.12
2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] git version: 33934938e0e95d534cebbaff656cde916b9c3573
2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] build info: Linux ip-10-229-66-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-04T02:41:16.911+0000 I CONTROL  [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "./data", engine: "wiredTiger" }, systemLog: { destination: "file", path: "/var/log/mongodb/mongod.log" } }
2018-04-04T02:41:16.933+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-04T02:41:16.938+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522809676:938331][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 45056: block header checksum of 1936028255 doesn't match expected checksum of 3000991456
2018-04-04T02:41:16.938+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522809676:938356][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
2018-04-04T02:41:16.938+0000 E STORAGE  [initandlisten] WiredTiger (-31804) [1522809676:938368][1713:0x7f41675eabc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-04-04T02:41:16.938+0000 I -        [initandlisten] Fatal Assertion 28558
2018-04-04T02:41:16.950+0000 I CONTROL  [initandlisten] 
 0xfa8242 0xf454d9 0xf29706 0xdca081 0x13ff32c 0x13ff4dd 0x13ff954 0x134832c 0x1348883 0x13454a2 0x13495ce 0x1365204 0x1390a5b 0x13fe6d9 0x13ca183 0x138dfc9 0xdb429b 0xdb1e70 0xab1dca 0x82f9b1 0x7fd249 0x7f4165eb7f45 0x82d6d7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BA8242","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"B454D9","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"B29706","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"9CA081"},{"b":"400000","o":"FFF32C","s":"__wt_eventv"},{"b":"400000","o":"FFF4DD","s":"__wt_err"},{"b":"400000","o":"FFF954","s":"__wt_panic"},{"b":"400000","o":"F4832C","s":"__wt_block_extlist_read"},{"b":"400000","o":"F48883","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"F454A2","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"F495CE"},{"b":"400000","o":"F65204","s":"__wt_btree_open"},{"b":"400000","o":"F90A5B","s":"__wt_conn_btree_get"},{"b":"400000","o":"FFE6D9","s":"__wt_session_get_btree"},{"b":"400000","o":"FCA183","s":"__wt_metadata_open"},{"b":"400000","o":"F8DFC9","s":"wiredtiger_open"},{"b":"400000","o":"9B429B","s":"_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb"},{"b":"400000","o":"9B1E70"},{"b":"400000","o":"6B1DCA","s":"_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs"},{"b":"400000","o":"42F9B1","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"3FD249","s":"main"},{"b":"7F4165E96000","o":"21F45","s":"__libc_start_main"},{"b":"400000","o":"42D6D7"}],"processInfo":{ "mongodbVersion" : "3.0.12", "gitVersion" : "33934938e0e95d534cebbaff656cde916b9c3573", "uname" : { "sysname" : "Linux", "release" : "3.13.0-92-generic", "version" : "#139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EBD85547D0C90260CB4C429A2A005349F1C70CB5" }, { "b" : "7FFD2E351000", "elfType" : 3, "buildId" : "7F14688F101A2ACE5CAD23DFBFBC918616651576" }, { "b" : "7F41671BE000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F4166F5F000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4166B83000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F416697B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F4166777000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F4166471000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F416625B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4165E96000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F41673DC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xfa8242]
 mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf454d9]
 mongod(_ZN5mongo13fassertFailedEi+0x66) [0xf29706]
 mongod(+0x9CA081) [0xdca081]
 mongod(__wt_eventv+0x49C) [0x13ff32c]
 mongod(__wt_err+0x8D) [0x13ff4dd]
 mongod(__wt_panic+0x24) [0x13ff954]
 mongod(__wt_block_extlist_read+0x6C) [0x134832c]
 mongod(__wt_block_extlist_read_avail+0x33) [0x1348883]
 mongod(__wt_block_checkpoint_load+0x202) [0x13454a2]
 mongod(+0xF495CE) [0x13495ce]
 mongod(__wt_btree_open+0xB54) [0x1365204]
 mongod(__wt_conn_btree_get+0x1AB) [0x1390a5b]
 mongod(__wt_session_get_btree+0x449) [0x13fe6d9]
 mongod(__wt_metadata_open+0x33) [0x13ca183]
 mongod(wiredtiger_open+0xD99) [0x138dfc9]
 mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x53B) [0xdb429b]
 mongod(+0x9B1E70) [0xdb1e70]
 mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x32A) [0xab1dca]
 mongod(_ZN5mongo13initAndListenEi+0x421) [0x82f9b1]
 mongod(main+0x159) [0x7fd249]
 libc.so.6(__libc_start_main+0xF5) [0x7f4165eb7f45]
 mongod(+0x42D6D7) [0x82d6d7]
-----  END BACKTRACE  -----
2018-04-04T02:41:16.950+0000 I -        [initandlisten] 
 
***aborting after fassert() failure



 Comments   
Comment by Kelsey Schubert [ 10/Apr/18 ]

Hi smallcar88,

Unfortunately, these errors indicate corruption across multiple files, which is most often caused by a faulty storage layer. In this situation, our best recommendation would be to resync the affected node or restore from a backup if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Martins Lui [ 06/Apr/18 ]

@kelsey.schubert

I have updated the issue

Comment by Martins Lui [ 06/Apr/18 ]

Thank you @Kelsey T Schubert for the files.
It does help for repairing the database. However, the repair process halts when repairing one of the collections (i.e. "luxify-scrap.listings_keys") , showing that one of the collection .wt is missing. Is there any way that I can keep some of the collections? I can see those important collections have been successfully re-indexed.

Much Appreciated

2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] MongoDB starting : pid=17454 port=27017 dbpath=./data 64-bit host=ip-172-31-24-153
2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] db version v3.0.12
2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] git version: 33934938e0e95d534cebbaff656cde916b9c3573
2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] build info: Linux ip-10-229-66-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2018-04-06T06:29:50.859+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "./data", engine: "wiredTiger" }, systemLog: { destination: "file", path: "/var/log/mongodb/mongod.log" } }
2018-04-06T06:29:50.880+0000 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2018-04-06T06:29:50.880+0000 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2018-04-06T06:29:50.962+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2018-04-06T06:29:50.965+0000 I STORAGE  [initandlisten] Repairing size cache
2018-04-06T06:29:50.965+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522996190:965957][17454:0x7f1002f78bc0], file:sizeStorer.wt, session.verify: read checksum error for 4096B block at offset 20480: block header checksum of 1477521906 doesn't match expected checksum of 1876314591
2018-04-06T06:29:50.965+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 0
2018-04-06T06:29:50.965+0000 I STORAGE  [initandlisten] Verify failed on uri table:sizeStorer. Running a salvage operation.
2018-04-06T06:29:50.966+0000 I STORAGE  [initandlisten] WiredTiger progress session.salvage 8
2018-04-06T06:29:50.966+0000 I STORAGE  [initandlisten] Repairing catalog metadata
2018-04-06T06:29:50.967+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2018-04-06T06:29:50.967+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:_mdb_catalog. Not salvaging.
2018-04-06T06:29:50.973+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-04-06T06:29:50.973+0000 I CONTROL  [initandlisten] 
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] 
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] 
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-04-06T06:29:50.976+0000 I CONTROL  [initandlisten] 
2018-04-06T06:29:50.980+0000 I STORAGE  [initandlisten] repairDatabase admin
2018-04-06T06:29:50.981+0000 I STORAGE  [initandlisten] Repairing collection admin.agendaJobs
2018-04-06T06:29:50.983+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2018-04-06T06:29:50.984+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-6--3245188848534206563. Not salvaging.
2018-04-06T06:29:51.000+0000 I INDEX    [initandlisten] build index on: admin.agendaJobs properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "admin.agendaJobs" }
2018-04-06T06:29:51.001+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.010+0000 I INDEX    [initandlisten] build index on: admin.agendaJobs properties: { v: 1, key: { name: 1, priority: -1, lockedAt: 1, nextRunAt: 1, disabled: 1 }, name: "findAndLockNextJobIndex1", ns: "admin.agendaJobs" }
2018-04-06T06:29:51.010+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.017+0000 I INDEX    [initandlisten] build index on: admin.agendaJobs properties: { v: 1, key: { name: 1, lockedAt: 1, priority: -1, nextRunAt: 1, disabled: 1 }, name: "findAndLockNextJobIndex2", ns: "admin.agendaJobs" }
2018-04-06T06:29:51.017+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.044+0000 I STORAGE  [initandlisten] repairDatabase local
2018-04-06T06:29:51.044+0000 I STORAGE  [initandlisten] Repairing collection local.startup_log
2018-04-06T06:29:51.045+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2018-04-06T06:29:51.045+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--4780682076133409313. Not salvaging.
2018-04-06T06:29:51.053+0000 I INDEX    [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2018-04-06T06:29:51.053+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.061+0000 I STORAGE  [initandlisten] repairDatabase luxify-scrap
2018-04-06T06:29:51.061+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.agendaJobs
2018-04-06T06:29:51.061+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 2
2018-04-06T06:29:51.061+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2--3245188848534206563. Not salvaging.
2018-04-06T06:29:51.069+0000 I INDEX    [initandlisten] build index on: luxify-scrap.agendaJobs properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.agendaJobs" }
2018-04-06T06:29:51.069+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.077+0000 I INDEX    [initandlisten] build index on: luxify-scrap.agendaJobs properties: { v: 1, key: { name: 1, priority: -1, lockedAt: 1, nextRunAt: 1, disabled: 1 }, name: "findAndLockNextJobIndex1", ns: "luxify-scrap.agendaJobs" }
2018-04-06T06:29:51.077+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.085+0000 I INDEX    [initandlisten] build index on: luxify-scrap.agendaJobs properties: { v: 1, key: { name: 1, lockedAt: 1, priority: -1, nextRunAt: 1, disabled: 1 }, name: "findAndLockNextJobIndex2", ns: "luxify-scrap.agendaJobs" }
2018-04-06T06:29:51.085+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.111+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.canepa
2018-04-06T06:29:51.111+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
2018-04-06T06:29:51.111+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 20
2018-04-06T06:29:51.112+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 30
2018-04-06T06:29:51.112+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 40
2018-04-06T06:29:51.112+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 50
2018-04-06T06:29:51.113+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 56
2018-04-06T06:29:51.113+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-14--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.120+0000 I INDEX    [initandlisten] build index on: luxify-scrap.canepa properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.canepa" }
2018-04-06T06:29:51.120+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.129+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.canepa_2016-08-15
2018-04-06T06:29:51.133+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
2018-04-06T06:29:51.136+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 20
2018-04-06T06:29:51.139+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 30
2018-04-06T06:29:51.142+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 40
2018-04-06T06:29:51.143+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 42
2018-04-06T06:29:51.143+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-8--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.150+0000 I INDEX    [initandlisten] build index on: luxify-scrap.canepa_2016-08-15 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.canepa_2016-08-15" }
2018-04-06T06:29:51.150+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.159+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.canepa_2016-08-16
2018-04-06T06:29:51.159+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
2018-04-06T06:29:51.160+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 20
2018-04-06T06:29:51.160+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 30
2018-04-06T06:29:51.160+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 40
2018-04-06T06:29:51.161+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 50
2018-04-06T06:29:51.161+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 56
2018-04-06T06:29:51.161+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-10--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.169+0000 I INDEX    [initandlisten] build index on: luxify-scrap.canepa_2016-08-16 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.canepa_2016-08-16" }
2018-04-06T06:29:51.169+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.178+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.europeanCars
2018-04-06T06:29:51.183+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 6
2018-04-06T06:29:51.183+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-6--4780682076133409313. Not salvaging.
2018-04-06T06:29:51.192+0000 I INDEX    [initandlisten] build index on: luxify-scrap.europeanCars properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.europeanCars" }
2018-04-06T06:29:51.192+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.201+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.europeanCars_2016-07-06
2018-04-06T06:29:51.204+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 5
2018-04-06T06:29:51.204+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--3245188848534206563. Not salvaging.
2018-04-06T06:29:51.211+0000 I INDEX    [initandlisten] build index on: luxify-scrap.europeanCars_2016-07-06 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.europeanCars_2016-07-06" }
2018-04-06T06:29:51.211+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.220+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.europeanCars_2016-07-21
2018-04-06T06:29:51.224+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 5
2018-04-06T06:29:51.224+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.232+0000 I INDEX    [initandlisten] build index on: luxify-scrap.europeanCars_2016-07-21 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.europeanCars_2016-07-21" }
2018-04-06T06:29:51.232+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.240+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings
2018-04-06T06:29:51.241+0000 E STORAGE  [initandlisten] WiredTiger (0) [1522996191:241202][17454:0x7f1002f78bc0], file:collection-6--2393506298145415295.wt, session.verify: read checksum error for 4096B block at offset 8171520: block header checksum of 671185013 doesn't match expected checksum of 2789663372
2018-04-06T06:29:51.241+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 0
2018-04-06T06:29:51.241+0000 I STORAGE  [initandlisten] Verify failed on uri table:collection-6--2393506298145415295. Running a salvage operation.
2018-04-06T06:29:51.241+0000 I STORAGE  [initandlisten] WiredTiger progress session.salvage 10
[Skipped...]
2018-04-06T06:29:51.310+0000 I STORAGE  [initandlisten] WiredTiger progress session.salvage 1626
2018-04-06T06:29:51.319+0000 I INDEX    [initandlisten] build index on: luxify-scrap.listings properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.listings" }
2018-04-06T06:29:51.319+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.411+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings_2016-08-16
2018-04-06T06:29:51.415+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
[Skipped]
2018-04-06T06:29:51.470+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 176
2018-04-06T06:29:51.470+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-12--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.478+0000 I INDEX    [initandlisten] build index on: luxify-scrap.listings_2016-08-16 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.listings_2016-08-16" }
2018-04-06T06:29:51.478+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.496+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings_2016-08-17
2018-04-06T06:29:51.500+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
[Skipped...]
2018-04-06T06:29:51.559+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 182
2018-04-06T06:29:51.559+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-16--2393506298145415295. Not salvaging.
2018-04-06T06:29:51.568+0000 I INDEX    [initandlisten] build index on: luxify-scrap.listings_2016-08-17 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.listings_2016-08-17" }
2018-04-06T06:29:51.568+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.585+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings_2017-01-19
2018-04-06T06:29:51.586+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
[Skipped the log]
2018-04-06T06:29:51.610+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 688
2018-04-06T06:29:51.610+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-5176612929081867590. Not salvaging.
2018-04-06T06:29:51.619+0000 I INDEX    [initandlisten] build index on: luxify-scrap.listings_2017-01-19 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.listings_2017-01-19" }
2018-04-06T06:29:51.619+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.657+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings_2017-02-8
2018-04-06T06:29:51.658+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 10
[Skipped]
2018-04-06T06:29:51.684+0000 I STORAGE  [initandlisten] WiredTiger progress session.verify 759
2018-04-06T06:29:51.684+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2-5176612929081867590. Not salvaging.
2018-04-06T06:29:51.693+0000 I INDEX    [initandlisten] build index on: luxify-scrap.listings_2017-02-8 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "luxify-scrap.listings_2017-02-8" }
2018-04-06T06:29:51.693+0000 I INDEX    [initandlisten] 	 building index using bulk method
2018-04-06T06:29:51.733+0000 I STORAGE  [initandlisten] Repairing collection luxify-scrap.listings_keys
2018-04-06T06:29:51.733+0000 E STORAGE  [initandlisten] WiredTiger (2) [1522996191:733709][17454:0x7f1002f78bc0], file:collection-10--7210790718269791336.wt, session.verify: ./data/collection-10--7210790718269791336.wt: No such file or directory
2018-04-06T06:29:51.733+0000 I STORAGE  [initandlisten] Verify failed on uri table:collection-10--7210790718269791336. Running a salvage operation.
2018-04-06T06:29:51.733+0000 E STORAGE  [initandlisten] WiredTiger (2) [1522996191:733911][17454:0x7f1002f78bc0], file:collection-10--7210790718269791336.wt, session.salvage: ./data/collection-10--7210790718269791336.wt: No such file or directory
2018-04-06T06:29:51.734+0000 I -        [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 176
2018-04-06T06:29:51.746+0000 I CONTROL  [initandlisten] 
 0xfa8242 0xf454d9 0xf29639 0x9481cf 0x94a14e 0x94cf06 0xc31bb4 0x82fe31 0x7fd249 0x7f1001845f45 0x82d6d7
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"BA8242","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"B454D9","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"B29639","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"5481CF","s":"_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE"},{"b":"400000","o":"54A14E","s":"_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE"},{"b":"400000","o":"54CF06","s":"_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb"},{"b":"400000","o":"831BB4","s":"_ZN5mongo14repairDatabaseEPNS_16OperationContextEPNS_13StorageEngineERKSsbb"},{"b":"400000","o":"42FE31","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"3FD249","s":"main"},{"b":"7F1001824000","o":"21F45","s":"__libc_start_main"},{"b":"400000","o":"42D6D7"}],"processInfo":{ "mongodbVersion" : "3.0.12", "gitVersion" : "33934938e0e95d534cebbaff656cde916b9c3573", "uname" : { "sysname" : "Linux", "release" : "3.13.0-92-generic", "version" : "#139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EBD85547D0C90260CB4C429A2A005349F1C70CB5" }, { "b" : "7FFF964B8000", "elfType" : 3, "buildId" : "7F14688F101A2ACE5CAD23DFBFBC918616651576" }, { "b" : "7F1002B4C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F10028ED000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F1002511000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F1002309000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F1002105000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F1001DFF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F1001BE9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F1001824000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F1002D6A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xfa8242]
 mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf454d9]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB9) [0xf29639]
 mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0xEF) [0x9481cf]
 mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x1FE) [0x94a14e]
 mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x176) [0x94cf06]
 mongod(_ZN5mongo14repairDatabaseEPNS_16OperationContextEPNS_13StorageEngineERKSsbb+0x704) [0xc31bb4]
 mongod(_ZN5mongo13initAndListenEi+0x8A1) [0x82fe31]
 mongod(main+0x159) [0x7fd249]
 libc.so.6(__libc_start_main+0xF5) [0x7f1001845f45]
 mongod(+0x42D6D7) [0x82d6d7]
-----  END BACKTRACE  -----
2018-04-06T06:29:51.746+0000 I -        [initandlisten] 
 
***aborting after invariant() failure

Comment by Kelsey Schubert [ 04/Apr/18 ]

Hi smallcar88,

Thank you for your report. I've attached a repair attempt, repair_attempt.tar.gz, of the files you provided. Please extract these files and replace them in your $dbpath. Please note that MongoDB 3.0 is no longer supported, and I strongly recommend that you upgrade to a more recent version.

Thank you,
Kelsey

Generated at Thu Feb 08 04:36:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.