[SERVER-35573] Help! WiredTiger has failed to open its metadata Created: 13/Jun/18  Updated: 15/Sep/18  Resolved: 14/Jun/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.20
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: loco Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: docker, envc, rpo, rps, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

unraid 6.4.1 + docker 17.09 + dockerimage: mongo


Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair-attempt-2.tar.gz     File repair-attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

After an unexpected shutdown, mongodb does not start properly. 

I have tried the --repair run with mongo:3.2 and mongo:3.6, But no use.

 

Here is the console log:

2018-06-13T06:32:41.085+0000 I CONTROL [initandlisten] MongoDB starting : pid=13 port=27017 dbpath=/data/db 64-bit host=029732e706f2
2018-06-13T06:32:41.085+0000 I CONTROL [initandlisten] db version v3.2.20
2018-06-13T06:32:41.085+0000 I CONTROL [initandlisten] git version: a7a144f40b70bfe290906eb33ff2714933544af8
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] allocator: tcmalloc
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] modules: none
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] build environment:
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] distmod: debian81
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] distarch: x86_64
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] target_arch: x86_64
2018-06-13T06:32:41.086+0000 I CONTROL [initandlisten] options: \{ repair: true }
2018-06-13T06:32:41.110+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-06-13T06:32:41.110+0000 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2018-06-13T06:32:41.110+0000 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=46G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2018-06-13T06:32:41.121+0000 E STORAGE [initandlisten] WiredTiger (-31802) [1528871561:121395][13:0x147775671cc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2018-06-13T06:32:41.121+0000 E STORAGE [initandlisten] WiredTiger (0) [1528871561:121444][13:0x147775671cc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2018-06-13T06:32:41.121+0000 E STORAGE [initandlisten] WiredTiger (0) [1528871561:121455][13:0x147775671cc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2018-06-13T06:32:41.121+0000 E STORAGE [initandlisten] WiredTiger (0) [1528871561:121469][13:0x147775671cc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2018-06-13T06:32:41.122+0000 I - [initandlisten] Assertion: 28718:-31802: WT_ERROR: non-specific WiredTiger error
2018-06-13T06:32:41.122+0000 I STORAGE [initandlisten] exception in initAndListen: 28718 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2018-06-13T06:32:41.122+0000 I CONTROL [initandlisten] dbexit: rc: 100



 Comments   
Comment by Nick Brewer [ 14/Jun/18 ]

Glad to hear it's working!

Regards,
Nick

Comment by loco [ 14/Jun/18 ]

It's started working!! Thank you, Nick.

Comment by Nick Brewer [ 14/Jun/18 ]

locoz666666

I've tried another repair attempt. The results are attached - let us know if these work for you.

Thanks,
Nick

repair-attempt-2.tar.gz

Comment by loco [ 14/Jun/18 ]

I tried to replace the file, But it doesn't look very good...

console log:
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=60de07a59630
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] db version v3.6.5
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] git version: a20ecd3e3a174162052ff99913bc2ca9a839d618
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] allocator: tcmalloc
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] modules: none
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] build environment:
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] distmod: debian81
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] distarch: x86_64
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] target_arch: x86_64
2018-06-14T10:46:04.537+0800 I CONTROL [initandlisten] options: { net:

{ bindIpAll: true }

}
2018-06-14T10:46:04.538+0800 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-06-14T10:46:04.538+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=39783M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-06-14T10:46:05.257+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:257710][1:0x14d75b291a00], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 3308844398 doesn't match expected checksum of 357133071
2018-06-14T10:46:05.257+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:257932][1:0x14d75b291a00], file:WiredTiger.wt, connection: {12288, 4096, 357133071}: (chunk 1 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 00 00 00 2d 00 00 00 01 00 00 00 00 10 00 00 6e f5 38 c5 01 00 00 00 e2 f5 1a 80 e2 0f c0 cf c0 e2 bf c0 e2 5f c0 e3 01 8f c0 e2 1f c0 e3 02 1f c0 e2 2f c0 e3 02 cf c0 e2 3f c0 e3 03 8f c0 e2 cf c0 80 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2018-06-14T10:46:05.258+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:258099][1:0x14d75b291a00], file:WiredTiger.wt, connection: {12288, 4096, 357133071}: (chunk 2 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2018-06-14T10:46:05.258+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:258254][1:0x14d75b291a00], file:WiredTiger.wt, connection: {12288, 4096, 357133071}: (chunk 3 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2018-06-14T10:46:05.258+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:258408][1:0x14d75b291a00], file:WiredTiger.wt, connection: {12288, 4096, 357133071}: (chunk 4 of 4): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2018-06-14T10:46:05.258+0800 E STORAGE [initandlisten] WiredTiger error (0) [1528944365:258428][1:0x14d75b291a00], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2018-06-14T10:46:05.258+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1528944365:258446][1:0x14d75b291a00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2018-06-14T10:46:05.258+0800 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2018-06-14T10:46:05.258+0800 F - [initandlisten]

***aborting after fassert() failure

2018-06-14T10:46:05.283+0800 F - [initandlisten] Got signal: 6 (Aborted).

0x5584bb720aa1 0x5584bb71fcb9 0x5584bb72019d 0x14d759cd5890 0x14d759950067 0x14d759951448 0x5584b9e7d955 0x5584b9f4ab8e 0x5584b9fb7261 0x5584b9e1b389 0x5584b9e1b6b2 0x5584ba0664c7 0x5584ba08273f 0x5584ba082c8b 0x5584ba07f205 0x5584ba065bc7 0x5584b9fdd007 0x5584b9f5d1e6 0x5584b9fb5abd 0x5584b9fb6092 0x5584b9fb62fc 0x5584ba023e46 0x5584b9fb16f8 0x5584b9f7ae4e 0x5584b9f7af2b 0x5584b9f5bbb2 0x5584b9f2e17a 0x5584b9f125d9 0x5584ba106f50 0x5584b9e172b0 0x5584b9ef18eb 0x5584b9e7f619 0x14d75993cb45 0x5584b9ee104f
----- BEGIN BACKTRACE -----

{"backtrace":[

{"b":"5584B94EF000","o":"2231AA1","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"5584B94EF000","o":"2230CB9"},{"b":"5584B94EF000","o":"223119D"},{"b":"14D759CC6000","o":"F890"},{"b":"14D75991B000","o":"35067","s":"gsignal"},{"b":"14D75991B000","o":"36448","s":"abort"},

{"b":"5584B94EF000","o":"98E955","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"}

,{"b":"5584B94EF000","o":"A5BB8E"},{"b":"5584B94EF000","o":"AC8261"},

{"b":"5584B94EF000","o":"92C389","s":"__wt_err"}

,

{"b":"5584B94EF000","o":"92C6B2","s":"__wt_panic"}

,

{"b":"5584B94EF000","o":"B774C7","s":"__wt_block_read_off"}

,

{"b":"5584B94EF000","o":"B9373F","s":"__wt_block_extlist_read"}

,

{"b":"5584B94EF000","o":"B93C8B","s":"__wt_block_extlist_read_avail"}

,

{"b":"5584B94EF000","o":"B90205","s":"__wt_block_checkpoint_load"}

,{"b":"5584B94EF000","o":"B76BC7"},

{"b":"5584B94EF000","o":"AEE007","s":"__wt_btree_open"}

,

{"b":"5584B94EF000","o":"A6E1E6","s":"__wt_conn_dhandle_open"}

,

{"b":"5584B94EF000","o":"AC6ABD","s":"__wt_session_get_dhandle"}

,

{"b":"5584B94EF000","o":"AC7092","s":"__wt_session_get_dhandle"}

,

{"b":"5584B94EF000","o":"AC72FC","s":"__wt_session_get_btree_ckpt"}

,

{"b":"5584B94EF000","o":"B34E46","s":"__wt_curfile_open"}

,{"b":"5584B94EF000","o":"AC26F8"},

{"b":"5584B94EF000","o":"A8BE4E","s":"__wt_metadata_cursor_open"}

,

{"b":"5584B94EF000","o":"A8BF2B","s":"__wt_metadata_cursor"}

,{"b":"5584B94EF000","o":"A6CBB2","s":"wiredtiger_open"},

{"b":"5584B94EF000","o":"A3F17A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"}

,{"b":"5584B94EF000","o":"A235D9"},{"b":"5584B94EF000","o":"C17F50","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5584B94EF000","o":"9282B0"},

{"b":"5584B94EF000","o":"A028EB","s":"_ZN5mongo11mongoDbMainEiPPcS1_"}

,{"b":"5584B94EF000","o":"990619","s":"main"},{"b":"14D75991B000","o":"21B45","s":"__libc_start_main"},{"b":"5584B94EF000","o":"9F204F"}],"processInfo":{ "mongodbVersion" : "3.6.5", "gitVersion" : "a20ecd3e3a174162052ff99913bc2ca9a839d618", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "4.14.16-unRAID", "version" : "#1 SMP PREEMPT Wed Jan 31 08:51:39 PST 2018", "machine" : "x86_64" }

, "somap" : [ { "b" : "5584B94EF000", "elfType" : 3, "buildId" : "CBBADA84B25F33AF76A5C7F23749CB9D9EECD010" }, { "b" : "7FFCBD9CD000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "00A6B9AA07EAC32796843CA84B76F07156216AF1" }, { "b" : "14D75AE64000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "14D75AA67000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "0C9DA403601A5EEA627AF96E1EB63DD22B8DC28B" }, { "b" : "14D75A806000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "EDE40F0BC2115063088BF442E0F2ED84BF76B11E" }, { "b" : "14D75A602000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "14D75A3FA000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "14D75A0F9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "14D759EE3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "14D759CC6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "14D75991B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "14D75B07B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5584bb720aa1]
mongod(+0x2230CB9) [0x5584bb71fcb9]
mongod(+0x223119D) [0x5584bb72019d]
libpthread.so.0(+0xF890) [0x14d759cd5890]
libc.so.6(gsignal+0x37) [0x14d759950067]
libc.so.6(abort+0x148) [0x14d759951448]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5584b9e7d955]
mongod(+0xA5BB8E) [0x5584b9f4ab8e]
mongod(+0xAC8261) [0x5584b9fb7261]
mongod(__wt_err+0x9D) [0x5584b9e1b389]
mongod(__wt_panic+0x33) [0x5584b9e1b6b2]
mongod(__wt_block_read_off+0x547) [0x5584ba0664c7]
mongod(__wt_block_extlist_read+0x8F) [0x5584ba08273f]
mongod(__wt_block_extlist_read_avail+0x2B) [0x5584ba082c8b]
mongod(__wt_block_checkpoint_load+0x275) [0x5584ba07f205]
mongod(+0xB76BC7) [0x5584ba065bc7]
mongod(__wt_btree_open+0xC27) [0x5584b9fdd007]
mongod(__wt_conn_dhandle_open+0x276) [0x5584b9f5d1e6]
mongod(__wt_session_get_dhandle+0xED) [0x5584b9fb5abd]
mongod(__wt_session_get_dhandle+0x6C2) [0x5584b9fb6092]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x5584b9fb62fc]
mongod(__wt_curfile_open+0x56) [0x5584ba023e46]
mongod(+0xAC26F8) [0x5584b9fb16f8]
mongod(__wt_metadata_cursor_open+0x6E) [0x5584b9f7ae4e]
mongod(__wt_metadata_cursor+0x4B) [0x5584b9f7af2b]
mongod(wiredtiger_open+0x1942) [0x5584b9f5bbb2]
mongod(ZN5mongo18WiredTigerKVEngineC1ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x85A) [0x5584b9f2e17a]
mongod(+0xA235D9) [0x5584b9f125d9]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x650) [0x5584ba106f50]
mongod(+0x9282B0) [0x5584b9e172b0]
mongod(ZN5mongo11mongoDbMainEiPPcS1+0x86B) [0x5584b9ef18eb]
mongod(main+0x9) [0x5584b9e7f619]
libc.so.6(__libc_start_main+0xF5) [0x14d75993cb45]
mongod(+0x9F204F) [0x5584b9ee104f]
----- END BACKTRACE -----

Comment by Nick Brewer [ 13/Jun/18 ]

Hi,

I've attached the files after a repair attempt. Would you please extract these files, substitute them for the current ones in your $dbpath, and let us know if it resolves the issue?

Thanks,

Nick

repair-attempt.tar.gz

Generated at Thu Feb 08 04:40:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.