[SERVER-44589] mongodb fails to start after outage Created: 12/Nov/19  Updated: 09/Dec/19  Resolved: 09/Dec/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.20
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Reg Dancy Assignee: Carl Champain (Inactive)
Resolution: Done Votes: 0
Labels: WiredTiger, repair
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian 9 OS


Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair_attempt.tar.gz    
Participants:

 Description   

 

dccls2rh:~ # /etc/init.d/mongod start
 [....] Starting database: mongod2019-11-07T12:01:31.736-0500 I CONTROL [main] ***** SERVER RESTARTED *****
 2019-11-07T12:01:31.743-0500 I CONTROL [initandlisten] MongoDB starting : pid=4763 port=27017 dbpath=/var/lib/mongodb 64-bit host=dwdccls2rh
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] db version v3.2.20
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] allocator: tcmalloc
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] modules: none
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] build environment:
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] distmod: debian71
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] distarch: x86_64
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] target_arch: x86_64
 2019-11-07T12:01:31.744-0500 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
 
{ bindIp: "10.32.13.226", port: 27017 }
 
, security: \{ authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal:
 
{ enabled: true }
 
}, systemLog: \{ destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
 2019-11-07T12:01:31.766-0500 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
 2019-11-07T12:01:31.767-0500 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
 2019-11-07T12:01:31.767-0500 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
 2019-11-07T12:01:31.767-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
 2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (0) [1573146091:789354][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: read checksum error for 4096B block at offset 12288: block header checksum of 1043026774 doesn't match expected checksum of 1182754036
 2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (0) [1573146091:789416][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: WiredTiger.wt: encountered an illegal file format or internal value
 2019-11-07T12:01:31.789-0500 E STORAGE [initandlisten] WiredTiger (-31804) [1573146091:789436][4763:0x7f33c6e1dc80], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
 2019-11-07T12:01:31.789-0500 I - [initandlisten] Fatal Assertion 28558
 2019-11-07T12:01:31.789-0500 I - [initandlisten]
  
 ***aborting after fassert() failure
  
  
 2019-11-07T12:01:31.807-0500 F - [initandlisten] Got signal: 6 (Aborted).
  
 0x133d2f2 0x133c219 0x133ca22 0x7f33c5b010a0 0x7f33c5797125 0x7f33c579a3a0 0x12b9732 0x109a833 0x1ac2338 0x1ac2535 0x1ac2703 0x19e50dc 0x19e5643 0x19e1ed9 0x19e68d9 0x19ffc05 0x1a3a68b 0x1ac0c0d 0x1ac1129 0x1ac134b 0x1a47e76 0x1abda9d 0x1a81e0f 0x1a81f0e 0x1a3631a 0x1081116 0x107d1f3 0xfa0cae 0x955710 0x958c0d 0x7f33c5783ead 0x9510c9
 ----- BEGIN BACKTRACE -----
 
{"backtrace":[\\{"b":"400000","o":"F3D2F2","s":"_ZN5mongo15printStackTraceERSo"}
 
,\{"b":"400000","o":"F3C219"},\{"b":"400000","o":"F3CA22"},\{"b":"7F33C5AF2000","o":"F0A0"},\{"b":"7F33C5765000","o":"32125","s":"gsignal"},\{"b":"7F33C5765000","o":"353A0","s":"abort"},{"b":"400000","o":"EB9732","s":"_ZN5mongo13fassertFailedEi"},\{"b":"400000","o":"C9A833"},{"b":"400000","o":"16C2338","s":"__wt_eventv"},{"b":"400000","o":"16C2535","s":"__wt_err"},{"b":"400000","o":"16C2703","s":"__wt_panic"},{"b":"400000","o":"15E50DC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15E5643","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15E1ED9","s":"__wt_block_checkpoint_load"},\{"b":"400000","o":"15E68D9"},{"b":"400000","o":"15FFC05","s":"__wt_btree_open"},{"b":"400000","o":"163A68B","s":"__wt_conn_btree_open"},{"b":"400000","o":"16C0C0D","s":"__wt_session_get_btree"},{"b":"400000","o":"16C1129","s":"__wt_session_get_btree"},{"b":"400000","o":"16C134B","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1647E76","s":"__wt_curfile_open"},\{"b":"400000","o":"16BDA9D"},{"b":"400000","o":"1681E0F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1681F0E","s":"__wt_metadata_cursor"},\{"b":"400000","o":"163631A","s":"wiredtiger_open"},\{"b":"400000","o":"C81116","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},\{"b":"400000","o":"C7D1F3"},\{"b":"400000","o":"BA0CAE","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},\{"b":"400000","o":"555710"},\{"b":"400000","o":"558C0D","s":"main"},{"b":"7F33C5765000","o":"1EEAD","s":"__libc_start_main"},\{"b":"400000","o":"5510C9"}],"processInfo":{ "mongodbVersion" : "3.2.20", "gitVersion" : "a7a144f40b70bfe290906eb33ff2714933544af8", "compiledModules" : [], "uname" :
 
{ "sysname" : "Linux", "release" : "3.2.0-6-amd64", "version" : "#1 SMP Debian 3.2.102-1", "machine" : "x86_64" }
 
, "somap" : [ \{ "elfType" : 2, "b" : "400000", "buildId" : "D879C8CFBD84A8F158AA3605759FB36ABDEB5366" }, \{ "b" : "7FFFC06FE000", "elfType" : 3, "buildId" : "7385007A08B4A88E926289F83C79E9A4D4561B82" }, \{ "b" : "7F33C69AC000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "30890712A16EDD0CDEDC66CF8E41E7E8010C4185" }, \{ "b" : "7F33C65B2000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3579FDA4D1E7D6281305E12E320EBBBDE1306D1A" }, \{ "b" : "7F33C63AA000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "CEA0BF1B6D309A394069A065AE4AE0813E5899F8" }, \{ "b" : "7F33C61A6000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "41E934315B95097DCEE271A02A0F390D07EF64AD" }, \{ "b" : "7F33C5F24000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "B62C3FA2DA9E617FF960A44873D761742BB17E90" }, \{ "b" : "7F33C5D0E000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "F980B1188708F8D8B5C35D185444AF4CB939AA1E" }, \{ "b" : "7F33C5AF2000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C86440A1C3888E3F3EA9ADDA2F5AC4CF2512DEFC" }, \{ "b" : "7F33C5765000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "8E96AC03BF6C01584DDC9750C326041E6218F0B8" }, \{ "b" : "7F33C6C0D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "91C1780EFA29E8E10441CEF90A00F1CE2D67AB2A" }, \{ "b" : "7F33C554E000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "1EFEB71FD4999C2307570D673A724EA4E1D85267" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x133d2f2]
 mongod(+0xF3C219) [0x133c219]
 mongod(+0xF3CA22) [0x133ca22]
 libpthread.so.0(+0xF0A0) [0x7f33c5b010a0]
 libc.so.6(gsignal+0x35) [0x7f33c5797125]
 libc.so.6(abort+0x180) [0x7f33c579a3a0]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12b9732]
 mongod(+0xC9A833) [0x109a833]
 mongod(__wt_eventv+0x440) [0x1ac2338]
 mongod(__wt_err+0x8D) [0x1ac2535]
 mongod(__wt_panic+0x2E) [0x1ac2703]
 mongod(__wt_block_extlist_read+0x6C) [0x19e50dc]
 mongod(__wt_block_extlist_read_avail+0x33) [0x19e5643]
 mongod(__wt_block_checkpoint_load+0x359) [0x19e1ed9]
 mongod(+0x15E68D9) [0x19e68d9]
 mongod(__wt_btree_open+0xBD5) [0x19ffc05]
 mongod(__wt_conn_btree_open+0x16B) [0x1a3a68b]
 mongod(__wt_session_get_btree+0xDD) [0x1ac0c0d]
 mongod(__wt_session_get_btree+0x5F9) [0x1ac1129]
 mongod(__wt_session_get_btree_ckpt+0xAB) [0x1ac134b]
 mongod(__wt_curfile_open+0x86) [0x1a47e76]
 mongod(+0x16BDA9D) [0x1abda9d]
 mongod(__wt_metadata_cursor_open+0x5F) [0x1a81e0f]
 mongod(__wt_metadata_cursor+0x7E) [0x1a81f0e]
 mongod(wiredtiger_open+0x191A) [0x1a3631a]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x5B6) [0x1081116]
 mongod(+0xC7D1F3) [0x107d1f3]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x59E) [0xfa0cae]
 mongod(+0x555710) [0x955710]
 mongod(main+0x15D) [0x958c0d]
 libc.so.6(__libc_start_main+0xFD) [0x7f33c5783ead]
 mongod(+0x5510C9) [0x9510c9]
 ----- END BACKTRACE -----

failed!



 Comments   
Comment by Carl Champain (Inactive) [ 09/Dec/19 ]

Hi reginald.dancy@comodo.com,

I'm sorry to hear that the repair attempt has failed. Upgrading to the latest version 4.2.1 and using a replica set should indeed reduce the risk of data corruption.
I'm going to close this ticket.

Thank you,
Carl

Comment by Reg Dancy [ 06/Dec/19 ]

Hi Carl,

I extracted the files in the proper Mongo directory. I tried to restart but it still fails.

I will notify the customer that we should upgrade beyond 3.2 at a later date.

Thanks!

Reg

Comment by Carl Champain (Inactive) [ 05/Dec/19 ]

Hi reginald.dancy@comodo.com,

I've attached a repair attempt of the files you provided as repair_attempt.tar.gz. Please extract these files, replace them in your $dbpath, and let us know if it resolves the issue.

Importantly, 3.2 is already end-of-lifed, which means it is unsupported by us. We strongly recommend you upgrade to a supported version and use a replica set to reduce the risk of data loss.

Thanks,
Carl
 

Comment by Reg Dancy [ 04/Dec/19 ]

WiredTiger.turtle

 

Hi Carl,

Sorry for the delay. Here are the requested files.

Regards,

Reg

Comment by Carl Champain (Inactive) [ 04/Dec/19 ]

Hi reginald.dancy@comodo.com,

Any updates on this issue?

Comment by Carl Champain (Inactive) [ 15/Nov/19 ]

Hi reginald.dancy@comodo.com,

Thanks for the report.
Can you please attach copies of the wiredTiger.wt and wiredTiger.turtle files? We can attempt a metadata-only repair effort using internal tools. Please keep in mind that this repair effort may not be successful.

Kind regards,
Carl
 

Comment by Reg Dancy [ 13/Nov/19 ]

Hi Ryan, yes I didn't see the selection for server. Thanks for placing it there.

Comment by Ryan Chipman [ 12/Nov/19 ]

hi reginald.dancy@comodo.com, looks like you meant to file this in the SERVER project instead of the TOOLS, so I've moved it there.

Generated at Thu Feb 08 05:06:24 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.