[SERVER-53606] WiredTiger error Created: 06/Jan/21  Updated: 21/May/21  Resolved: 21/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Raditya Iryandi Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-53607 WiredTiger error Closed
Operating System: ALL
Participants:

 Description   

mongod@node24511-aduankonten ~ $ mongod --repair --dbpath=/var/lib/mongo/
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] MongoDB starting : pid=7882 port=27017 dbpath=/var/lib/mongo/ 64-bit host=node24511-aduankonten.kilatiron.com
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] db version v3.6.8
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] git version: 6bc9ed599c3fa164703346a22bad17e33fa913e4
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] allocator: tcmalloc
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] modules: none
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] build environment:
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] distmod: rhel70
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] distarch: x86_64
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] target_arch: x86_64
2021-01-06T13:31:03.886+0700 I CONTROL [initandlisten] options: { repair: true, storage:

{ dbPath: "/var/lib/mongo/" }

}
2021-01-06T13:31:03.887+0700 I - [initandlisten] Detected data files in /var/lib/mongo/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-01-06T13:31:03.887+0700 I STORAGE [initandlisten]
2021-01-06T13:31:03.887+0700 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-01-06T13:31:03.887+0700 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-01-06T13:31:03.887+0700 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
2021-01-06T13:31:03.887+0700 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=7616M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-01-06T13:31:04.682+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:682596][7882:0x7f1913204b00], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1701273968 doesn't match expected checksum of 1955708714
2021-01-06T13:31:04.683+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:683044][7882:0x7f1913204b00], file:WiredTiger.wt, connection: {12288, 4096, 1955708714}: (chunk 1 of 4): 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 95 66 69 6c 65 3a 69 6e 64 65 78 2d 35 31 2d 32 38 34 33 30 33 32 35 38 35 35 38 33 36 38 35 32 31 37 2e 77 74 00 80 c3 96 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 75 6e 69 71 75 65 22 20 3a 20 74 72 75 65 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 66 69 6c 65 73 5f 69 64 22 20 3a 20 31 2c 20 22 6e 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 66 69 6c 65 73 5f 69 64 5f 31 5f 6e 5f 31 22 2c 20 22 6e 73 22 20 3a 20 22 61 64 75 61 6e 5f 64 65 76 2e 66 73 2e 63 68 75 6e 6b 73 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 32 3d 28 61 64 64 72 3d 22 30 31 38 31 38 31 65 34 62 39 64 63 34 36 31 31 38 32 38 31 65 34 66 32 65 30 39 64 61 30 38 30 38 30 38 30 38 30 38 30 38 30 65 32 31 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 32 2c 74 69 6d 65 3d 31 35 35 34 35 38 38 36 39 32 2c 73 69 7a 65 3d 38 31 39 32 2c 77 72 69 74 65 5f 67 65 6e 3d 32 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 36 2c 33 37 39 33 39 32 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 37 33 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f
2021-01-06T13:31:04.683+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:683458][7882:0x7f1913204b00], file:WiredTiger.wt, connection: {12288, 4096, 1955708714}: (chunk 2 of 4): 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 95 66 69 6c 65 3a 69 6e 64 65 78 2d 35 33 2d 32 38 34 33 30 33 32 35 38 35 35 38 33 36 38 35 32 31 37 2e 77 74 00 80 c3 69 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 5f 69 64 5f 22 2c 20 22 6e 73 22 20 3a 20 22 61 64 75 61 6e 5f 64 65 76 2e 61 64 75 61 6e 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 30 31 38 31 38 31 65 34 62 39 32 61 32 61 34 61 38 32 38 31 65 34 66 32 65 30 39 64 61 30 38 30 38 30 38 30 38 30 38 30 38 30 65 32 31 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 35 35 34 35 38 38 36 39 32 2c 73 69 7a 65 3d 38 31 39 32 2c 77 72 69 74 65 5f 67 65 6e 3d 32 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 36 2c 33 37 39 33 39 32 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64
2021-01-06T13:31:04.683+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:683783][7882:0x7f1913204b00], file:WiredTiger.wt, connection: {12288, 4096, 1955708714}: (chunk 3 of 4): 3d 37 35 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 95 66 69 6c 65 3a 69 6e 64 65 78 2d 35 35 2d 32 38 34 33 30 33 32 35 38 35 35 38 33 36 38 35 32 31 37 2e 77 74 00 80 c3 70 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 5f 69 64 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 5f 69 64 5f 22 2c 20 22 6e 73 22 20 3a 20 22 61 64 75 61 6e 5f 64 65 76 2e 6e 6f 74 69 66 69 63 61 74 69 6f 6e 22 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 3d 28 61 64 64 72 3d 22 30 31 38 31 38 31 65 34 34 38 64 64 65 31 30 36 38 32 38 31 65 34 66 32 65 30 39 64 61 30 38 30 38 30 38 30 38 30 38 30 38 30 65 32 31 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 31 2c 74 69 6d 65 3d 31 35 35 34 35 38 38 36 39 32 2c 73 69 7a 65 3d 38 31 39 32 2c 77 72 69 74 65 5f 67 65 6e 3d 32 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 36 2c 33 37 39 33 39 32 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d
2021-01-06T13:31:04.684+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:684118][7882:0x7f1913204b00], file:WiredTiger.wt, connection: {12288, 4096, 1955708714}: (chunk 4 of 4): 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 37 37 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 75 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 31 36 6b 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 30 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 35 4d 42 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 74 72 75 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 95 66 69 6c 65 3a 69 6e 64 65 78 2d 36 2d 2d 37 37 33 36 30 32 30 31 38 32 31 30 37 35 37 30 38 37 31 2e 77 74 00 80 c3 b4 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 38 2c 69 6e 66 6f 4f 62 6a 3d 7b 20 22 76 22 20 3a 20 32 2c 20 22 6b 65 79 22 20 3a 20 7b 20 22 6c 61 73 74 55 73 65 22 20 3a 20 31 20 7d 2c 20 22 6e 61 6d 65 22 20 3a 20 22 6c 73 69 64 54 54 4c 49 6e 64 65 78 22 2c 20 22 6e 73 22 20 3a 20 22 63 6f 6e 66 69 67 2e 73 79 73 74 65 6d 2e 73 65 73 73 69 6f 6e 73 22 2c 20 22 65 78 70 69 72 65 41 66 74 65 72 53 65 63 6f 6e 64 73 22 20 3a 20 31 38 30 30 20 7d 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 31 31 32 34 30 39 3d 28 61 64 64 72 3d 22 30 31 38 32 38 31 65 34 62 33 38 38 35 30 30 66 38 33 38 31 65 34 38 63 37 36 64 37 61 63 38 34 38 31 65 34 37 64 39 30 30 61 31 31 38 30 38 30 38 30 65 32 36 66 63 30 63 66 63 30 22 2c 6f 72 64 65 72 3d 31 31 32 34 30 39 2c 74 69 6d 65 3d 31 36 30 39 38 35 37 35 39 32 2c 73 69 7a 65 3d 38 31 39 32 2c 77 72 69 74 65 5f 67 65 6e 3d 32 31 38 34 38 36 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 33 39 39 2c 35 32 32 38 30
2021-01-06T13:31:04.684+0700 E STORAGE [initandlisten] WiredTiger error (0) [1609914664:684285][7882:0x7f1913204b00], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2021-01-06T13:31:04.684+0700 E STORAGE [initandlisten] WiredTiger error (-31804) [1609914664:684352][7882:0x7f1913204b00], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2021-01-06T13:31:04.684+0700 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2021-01-06T13:31:04.684+0700 F - [initandlisten]

***aborting after fassert() failure

2021-01-06T13:31:04.727+0700 F - [initandlisten] Got signal: 6 (Aborted).

0x558ce07fba61 0x558ce07fac79 0x558ce07fb15d 0x7f1911bd76d0 0x7f1911831277 0x7f1911832968 0x558cdef4c1d8 0x558cdf01a20e 0x558cdf087b41 0x558cdeee8531 0x558cdeee885a 0x558cdf137177 0x558cdf153c4f 0x558cdf15419b 0x558cdf150715 0x558cdf136877 0x558cdf0adea7 0x558cdf02c946 0x558cdf0863bd 0x558cdf08699a 0x558cdf086c0c 0x558cdf0f4786 0x558cdf081f88 0x558cdf04ae5e 0x558cdf04af3b 0x558cdf02b38b 0x558cdeffda3a 0x558cdefe1a94 0x558cdf1d8577 0x558cdeee428c 0x558cdefc095c 0x558cdef4de99 0x7f191181d445 0x558cdefb037f
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"558CDE5B7000","o":"2244A61","s":"_ZN5mongo15printStackTraceERSo"}

,{"b":"558CDE5B7000","o":"2243C79"},{"b":"558CDE5B7000","o":"224415D"},{"b":"7F1911BC8000","o":"F6D0"},{"b":"7F19117FB000","o":"36277","s":"gsignal"},{"b":"7F19117FB000","o":"37968","s":"abort"},{"b":"558CDE5B7000","o":"9951D8","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"558CDE5B7000","o":"A6320E"},{"b":"558CDE5B7000","o":"AD0B41"},{"b":"558CDE5B7000","o":"931531","s":"wt_err"},{"b":"558CDE5B7000","o":"93185A","s":"wt_panic"},{"b":"558CDE5B7000","o":"B80177","s":"wt_block_read_off"},{"b":"558CDE5B7000","o":"B9CC4F","s":"wt_block_extlist_read"},{"b":"558CDE5B7000","o":"B9D19B","s":"wt_block_extlist_read_avail"},{"b":"558CDE5B7000","o":"B99715","s":"wt_block_checkpoint_load"},{"b":"558CDE5B7000","o":"B7F877"},{"b":"558CDE5B7000","o":"AF6EA7","s":"wt_btree_open"},{"b":"558CDE5B7000","o":"A75946","s":"wt_conn_dhandle_open"},{"b":"558CDE5B7000","o":"ACF3BD","s":"wt_session_get_dhandle"},{"b":"558CDE5B7000","o":"ACF99A","s":"wt_session_get_dhandle"},{"b":"558CDE5B7000","o":"ACFC0C","s":"wt_session_get_btree_ckpt"},{"b":"558CDE5B7000","o":"B3D786","s":"wt_curfile_open"},{"b":"558CDE5B7000","o":"ACAF88"},{"b":"558CDE5B7000","o":"A93E5E","s":"wt_metadata_cursor_open"},{"b":"558CDE5B7000","o":"A93F3B","s":"wt_metadata_cursor"},{"b":"558CDE5B7000","o":"A7438B","s":"wiredtiger_open"},{"b":"558CDE5B7000","o":"A46A3A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"558CDE5B7000","o":"A2AA94"},{"b":"558CDE5B7000","o":"C21577","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"558CDE5B7000","o":"92D28C"},{"b":"558CDE5B7000","o":"A0995C","s":"_ZN5mongo11mongoDbMainEiPPcS1"},{"b":"558CDE5B7000","o":"996E99","s":"main"},{"b":"7F19117FB000","o":"22445","s":"__libc_start_main"},{"b":"558CDE5B7000","o":"9F937F"}],"processInfo":{ "mongodbVersion" : "3.6.8", "gitVersion" : "6bc9ed599c3fa164703346a22bad17e33fa913e4", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "3.10.0-957.12.2.vz7.96.21", "version" : "#1 SMP Thu Jun 27 15:10:55 MSK 2019", "machine" : "x86_64" }

, "somap" : [ { "b" : "558CDE5B7000", "elfType" : 3, "buildId" : "1081626D38759361CE9DE8B16F4E62C7005B55E5" }, { "b" : "7FFC08279000", "elfType" : 3, "buildId" : "3FF7157D434F710A3397FDC0EAE6A78DBD299221" }, { "b" : "7F1912DDB000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F191297A000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7F1912708000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7F1912504000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F19122FC000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F1911FFA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F1911DE4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F1911BC8000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F19117FB000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F1912FF4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F19115E5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F1911398000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "16FE0DC6CEFC5F444BC876516D02EFE9CC2D432F" }, { "b" : "7F19110B0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D1CD1B94855A85FBC735C745DB39BC096F7D8CC3" }, { "b" : "7F1910EAC000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2C7EF64EF0C5AF8BCFA8F9E628E5605A7D8C52D3" }, { "b" : "7F1910C79000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A20F715C514B3EA873F4CC77D585A50CB670E266" }, { "b" : "7F1910A6B000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "9F82B06CE44724A8ACE827A2C95A9A76619EA314" }, { "b" : "7F1910867000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F1910640000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "1B0C0DC4F9054C5922082921C1447739F27AC8F4" }, { "b" : "7F19103DE000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x558ce07fba61]
mongod(+0x2243C79) [0x558ce07fac79]
mongod(+0x224415D) [0x558ce07fb15d]
libpthread.so.0(+0xF6D0) [0x7f1911bd76d0]
libc.so.6(gsignal+0x37) [0x7f1911831277]
libc.so.6(abort+0x148) [0x7f1911832968]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x558cdef4c1d8]
mongod(+0xA6320E) [0x558cdf01a20e]
mongod(+0xAD0B41) [0x558cdf087b41]
mongod(__wt_err+0x9D) [0x558cdeee8531]
mongod(__wt_panic+0x33) [0x558cdeee885a]
mongod(__wt_block_read_off+0x547) [0x558cdf137177]
mongod(__wt_block_extlist_read+0x8F) [0x558cdf153c4f]
mongod(__wt_block_extlist_read_avail+0x2B) [0x558cdf15419b]
mongod(__wt_block_checkpoint_load+0x275) [0x558cdf150715]
mongod(+0xB7F877) [0x558cdf136877]
mongod(__wt_btree_open+0xC27) [0x558cdf0adea7]
mongod(__wt_conn_dhandle_open+0x276) [0x558cdf02c946]
mongod(__wt_session_get_dhandle+0xED) [0x558cdf0863bd]
mongod(__wt_session_get_dhandle+0x6CA) [0x558cdf08699a]
mongod(__wt_session_get_btree_ckpt+0x14C) [0x558cdf086c0c]
mongod(__wt_curfile_open+0x56) [0x558cdf0f4786]
mongod(+0xACAF88) [0x558cdf081f88]
mongod(__wt_metadata_cursor_open+0x6E) [0x558cdf04ae5e]
mongod(__wt_metadata_cursor+0x4B) [0x558cdf04af3b]
mongod(wiredtiger_open+0x1A0B) [0x558cdf02b38b]
mongod(ZN5mongo18WiredTigerKVEngineC1ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xC1A) [0x558cdeffda3a]
mongod(+0xA2AA94) [0x558cdefe1a94]
mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x637) [0x558cdf1d8577]
mongod(+0x92D28C) [0x558cdeee428c]
mongod(ZN5mongo11mongoDbMainEiPPcS1+0x86C) [0x558cdefc095c]
mongod(main+0x9) [0x558cdef4de99]
libc.so.6(__libc_start_main+0xF5) [0x7f191181d445]
mongod(+0x9F937F) [0x558cdefb037f]
----- END BACKTRACE -----
Aborted



 Comments   
Comment by Eric Sedor [ 21/May/21 ]

snkfnk@gmail.com, I am going to close this ticket for now. But if you are able to provide the information I requested here, we would be happy to reopen.

Sincerely,
Eric

Comment by Eric Sedor [ 15/Mar/21 ]

Hi,

We still need additional information to examine further. If this is still an issue for you, would you please provide the above?

Thanks,
Eric

Comment by Eric Sedor [ 06/Jan/21 ]

Hello, this error message leads us to suspect some form of physical corruption. Please make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath.

The ideal resolution is to perform a clean resync from an unaffected node.

You can also try mongod --repair using the latest version of MongoDB. In the event the --repair operation is unsuccessful, then please also provide:

  • A timeline of issues leading up to running --repair
  • The logs leading up to the first occurrence of any issue
  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.
Generated at Thu Feb 08 05:31:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.