[SERVER-77820] file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: Created: 05/Jun/23  Updated: 29/Jun/23  Resolved: 29/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Volodymyr Victorovich Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log    
Operating System: ALL
Participants:

 Description   

Hi,

We had a major power outage and it knocked down our mongo db running in docker as part of a stack. Now it won't come back up. Mongo had it's own volume mounted to /data/db.

 

I also ran this command "mongod --repair --dbpath /data/db" 

 

2023-06-05T20:58:32.000+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1685995112:622][318:0x7f9cdcc4fac0], file:WiredTiger.wt, connection: {110592, 4096, 640250159}: (chunk 4 of 4): 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 33 39 2d 32 33 39 33 38 30 32 38 31 38 32 34 39 33 31 34 34 39 33 2e 77 74 00 80 c3 24 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 32 3d 28 61 64 64 72 3d 22 30 31 38 37 38 31 65 34 31 39 36 33 32 35 39 65 38 38 38 31 65 34 38 37 63 65 63 65 65 65 38 30 38 30 38 30 38 30 38 30 38 30 65 32 37 66 63 30 65 32 34 66 63 30 22 2c 6f 72 64 65 72 3d 32 2c 74 69 6d 65 3d 31 36 37 35 32 31 30 32 33 39 2c 73 69 7a 65 3d 33 32 37 36 38 2c 77 72 69 74 65 5f 67 65 6e 3d 34 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 31 35 32 35 37 37 32 38 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 34 33 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65
2023-06-05T20:58:32.000+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1685995112:720][318:0x7f9cdcc4fac0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2023-06-05T20:58:32.000+0100 E STORAGE  [initandlisten] WiredTiger error (-31804) [1685995112:828][318:0x7f9cdcc4fac0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2023-06-05T20:58:32.000+0100 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2023-06-05T20:58:32.001+0100 F -        [initandlisten]
***aborting after fassert() failure
 
2023-06-05T20:58:32.017+0100 F -        [initandlisten] Got signal: 6 (Aborted).
 

 



 Comments   
Comment by Chris Kelly [ 29/Jun/23 ]

MongoDB 3.6 reached end of life in April 2021 and is no longer supported. The oldest supported version of MongoDB is currently 4.4. I am going to close this ticket, but leave some final remarks that may be of assistance.

If --repair is failing on 3.6, there have been updates made to it over the years, and there is some possibility --repair will work on a newer version of MongoDB. Failing that, you should first consider resyncing the affected node from a good node.

To avoid a problem like this in the future, it is our strong recommendation to:

Comment by Volodymyr Victorovich [ 07/Jun/23 ]

The entire log is in the file in the mongod.log

mongod --dbpath . --repair
^^

// 2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] MongoDB starting : pid=842 port=27017 dbpath=/config/data/db/ 64-bit host=node01
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] db version v3.6.8
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] modules: none
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] build environment:
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-07T08:09:16.326+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/config/data/db/" } }
2023-06-07T08:09:16.329+0100 I STORAGE  [initandlisten] exception in initAndListen: DBPathInUse: Unable to lock the lock file: /config/data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /config/data/db/ directory, terminating
2023-06-07T08:09:16.329+0100 I CONTROL  [initandlisten] now exiting
2023-06-07T08:09:16.329+0100 I CONTROL  [initandlisten] shutting down with code:100
root@node01:/usr/lib/unifi#
root@node01:/usr/lib/unifi#
root@node01:/usr/lib/unifi#
root@node01:/usr/lib/unifi# cd /config/data/db/
root@node01:/config/data/db# mongod --dbpath . --repair
2023-06-07T08:10:37.575+0100 I CONTROL  [initandlisten] MongoDB starting : pid=943 port=27017 dbpath=. 64-bit host=node01
2023-06-07T08:10:37.575+0100 I CONTROL  [initandlisten] db version v3.6.8
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] git version: 8e540c0b6db93ce994cc548f000900bdc740f80a
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] modules: none
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] build environment:
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-07T08:10:37.576+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "." } }
2023-06-07T08:10:37.577+0100 I -        [initandlisten] Detected data files in . created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-07T08:10:37.577+0100 I STORAGE  [initandlisten]
2023-06-07T08:10:37.577+0100 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2023-06-07T08:10:37.577+0100 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2023-06-07T08:10:37.577+0100 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2023-06-07T08:10:37.577+0100 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=2941M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),cache_cursors=false,compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-07T08:10:38.224+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:224910][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 110592: block header checksum of 1852123197 doesn't match expected checksum of 640250159
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:225083][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: {110592, 4096, 640250159}: (chunk 1 of 4): 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 32 37 2d 32 33 39 33 38 30 32 38 31 38 32 34 39 33 31 34 34 39 33 2e 77 74 00 80 c3 3d 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 34 31 31 30 3d 28 61 64 64 72 3d 22 30 31 39 31 38 31 65 34 31 34 32 35 39 30 61 33 39 32 38 31 65 34 61 63 39 32 39 34 38 65 39 33 38 31 65 34 34 39 34 37 35 61 33 37 38 30 38 30 38 30 65 33 35 65 34 66 63 30 65 33 32 39 61 66 63 30 22 2c 6f 72 64 65 72 3d 34 31 31 30 2c 74 69 6d 65 3d 31 36 37 37 33 35 38 32 30 33 2c 73 69 7a 65 3d 32 37 34 34 33 32 30 2c 77 72 69 74 65 5f 67 65 6e 3d 31 38 38 34 30 39 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 37 34 38 34 30 37 30 34 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 33 31 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:225204][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: {110592, 4096, 640250159}: (chunk 2 of 4): 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 33 31 2d 32 33 39 33 38 30 32 38 31 38 32 34 39 33 31 34 34 39 33 2e 77 74 00 80 c3 39 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 36 33 35 3d 28 61 64 64 72 3d 22 30 31 38 36 38 31 65 34 38 35 39 33 62 34 66 33 61 63 38 31 65 34 32 63 65 39 33 38 37 36 62 37 38 31 65 34 30 32 35 65 62 63 62 65 38 30 38 30 38 30 65 33 31 38 64 66 63 30 65 33 31 37 31 66 63 30 22 2c 6f 72 64 65 72 3d 36 33 35 2c 74 69 6d 65 3d 31 36 37 37 33 35 35 38 30 30 2c 73 69 7a 65 3d 31 35 32 37 38 30 38 2c 77 72 69 74 65 5f 67 65 6e 3d 36 30 36 39 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 37 34 36 39 32 36 30 38 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 33 35 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:225306][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: {110592, 4096, 640250159}: (chunk 3 of 4): 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 33 35 2d 32 33 39 33 38 30 32 38 31 38 32 34 39 33 31 34 34 39 33 2e 77 74 00 80 c3 35 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 33 38 3d 28 61 64 64 72 3d 22 30 31 38 36 38 31 65 34 61 30 34 30 63 33 31 32 38 63 38 31 65 34 37 61 66 30 66 63 36 37 39 34 38 31 65 34 61 39 65 63 37 30 31 31 38 30 38 30 38 30 65 33 30 37 64 66 63 30 65 33 30 36 66 66 63 30 22 2c 6f 72 64 65 72 3d 33 38 2c 74 69 6d 65 3d 31 36 37 37 32 38 33 38 35 36 2c 73 69 7a 65 3d 34 37 31 30 34 30 2c 77 72 69 74 65 5f 67 65 6e 3d 31 38 33 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 37 31 39 30 32 35 39 32 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 33 39 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:225407][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: {110592, 4096, 640250159}: (chunk 4 of 4): 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 3d 66 61 6c 73 65 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65 73 73 69 6f 6e 5f 6d 69 6e 3d 34 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 6d 69 6e 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 64 65 65 70 65 6e 5f 70 65 72 5f 63 68 69 6c 64 3d 30 2c 73 70 6c 69 74 5f 70 63 74 3d 39 30 2c 76 61 6c 75 65 5f 66 6f 72 6d 61 74 3d 75 2c 76 65 72 73 69 6f 6e 3d 28 6d 61 6a 6f 72 3d 31 2c 6d 69 6e 6f 72 3d 31 29 00 ad 66 69 6c 65 3a 63 6f 6c 6c 65 63 74 69 6f 6e 2d 32 33 39 2d 32 33 39 33 38 30 32 38 31 38 32 34 39 33 31 34 34 39 33 2e 77 74 00 80 c3 24 61 63 63 65 73 73 5f 70 61 74 74 65 72 6e 5f 68 69 6e 74 3d 6e 6f 6e 65 2c 61 6c 6c 6f 63 61 74 69 6f 6e 5f 73 69 7a 65 3d 34 4b 42 2c 61 70 70 5f 6d 65 74 61 64 61 74 61 3d 28 66 6f 72 6d 61 74 56 65 72 73 69 6f 6e 3d 31 29 2c 61 73 73 65 72 74 3d 28 63 6f 6d 6d 69 74 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 2c 72 65 61 64 5f 74 69 6d 65 73 74 61 6d 70 3d 6e 6f 6e 65 29 2c 62 6c 6f 63 6b 5f 61 6c 6c 6f 63 61 74 69 6f 6e 3d 62 65 73 74 2c 62 6c 6f 63 6b 5f 63 6f 6d 70 72 65 73 73 6f 72 3d 73 6e 61 70 70 79 2c 63 61 63 68 65 5f 72 65 73 69 64 65 6e 74 3d 66 61 6c 73 65 2c 63 68 65 63 6b 70 6f 69 6e 74 3d 28 57 69 72 65 64 54 69 67 65 72 43 68 65 63 6b 70 6f 69 6e 74 2e 32 3d 28 61 64 64 72 3d 22 30 31 38 37 38 31 65 34 31 39 36 33 32 35 39 65 38 38 38 31 65 34 38 37 63 65 63 65 65 65 38 30 38 30 38 30 38 30 38 30 38 30 65 32 37 66 63 30 65 32 34 66 63 30 22 2c 6f 72 64 65 72 3d 32 2c 74 69 6d 65 3d 31 36 37 35 32 31 30 32 33 39 2c 73 69 7a 65 3d 33 32 37 36 38 2c 77 72 69 74 65 5f 67 65 6e 3d 34 29 29 2c 63 68 65 63 6b 70 6f 69 6e 74 5f 6c 73 6e 3d 28 31 2c 31 35 32 35 37 37 32 38 29 2c 63 68 65 63 6b 73 75 6d 3d 6f 6e 2c 63 6f 6c 6c 61 74 6f 72 3d 2c 63 6f 6c 75 6d 6e 73 3d 2c 64 69 63 74 69 6f 6e 61 72 79 3d 30 2c 65 6e 63 72 79 70 74 69 6f 6e 3d 28 6b 65 79 69 64 3d 2c 6e 61 6d 65 3d 29 2c 66 6f 72 6d 61 74 3d 62 74 72 65 65 2c 68 75 66 66 6d 61 6e 5f 6b 65 79 3d 2c 68 75 66 66 6d 61 6e 5f 76 61 6c 75 65 3d 2c 69 64 3d 32 34 33 2c 69 67 6e 6f 72 65 5f 69 6e 5f 6d 65 6d 6f 72 79 5f 63 61 63 68 65 5f 73 69 7a 65 3d 66 61 6c 73 65 2c 69 6e 74 65 72 6e 61 6c 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 6d 61 78 3d 30 2c 69 6e 74 65 72 6e 61 6c 5f 6b 65 79 5f 74 72 75 6e 63 61 74 65 3d 74 72 75 65 2c 69 6e 74 65 72 6e 61 6c 5f 70 61 67 65 5f 6d 61 78 3d 34 4b 42 2c 6b 65 79 5f 66 6f 72 6d 61 74 3d 71 2c 6b 65 79 5f 67 61 70 3d 31 30 2c 6c 65 61 66 5f 69 74 65 6d 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 6b 65 79 5f 6d 61 78 3d 30 2c 6c 65 61 66 5f 70 61 67 65 5f 6d 61 78 3d 33 32 4b 42 2c 6c 65 61 66 5f 76 61 6c 75 65 5f 6d 61 78 3d 36 34 4d 42 2c 6c 6f 67 3d 28 65 6e 61 62 6c 65 64 3d 74 72 75 65 29 2c 6d 65 6d 6f 72 79 5f 70 61 67 65 5f 6d 61 78 3d 31 30 6d 2c 6f 73 5f 63 61 63 68 65 5f 64 69 72 74 79 5f 6d 61 78 3d 30 2c 6f 73 5f 63 61 63 68 65 5f 6d 61 78 3d 30 2c 70 72 65 66 69 78 5f 63 6f 6d 70 72 65
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (0) [1686121838:225438][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 302)
2023-06-07T08:10:38.225+0100 E STORAGE  [initandlisten] WiredTiger error (-31804) [1686121838:225457][943:0x7f5f08527ac0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
2023-06-07T08:10:38.225+0100 F -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2023-06-07T08:10:38.225+0100 F -        [initandlisten]***aborting after fassert() failure
2023-06-07T08:10:38.238+0100 F -        [initandlisten] Got signal: 6 (Aborted). 0x5639b8aad5fe 0x5639b8aaceef 0x5639b8aacf8c 0x7f5f087ed3c0 0x7f5f0862a03b 0x7f5f08609859 0x5639b7218fdd 0x5639b6fc0ace 0x5639b7355429 0x5639b6fcb0a8 0x5639b6fcb408 0x5639b6fd71d6 0x5639b741e644 0x5639b741eb8c 0x5639b741b776 0x5639b7401e68 0x5639b737a1ab 0x5639b72fb355 0x5639b7353cfe 0x5639b73542f0 0x5639b73544f4 0x5639b73c45b6 0x5639b734fa3f 0x5639b73174e2 0x5639b73175cf 0x5639b72f9ff7 0x5639b72cd438 0x5639b72b28f9 0x5639b7496386 0x5639b728c659 0x5639b728fb83 0x5639b721d99d 0x7f5f0860b0b3 0x5639b727a0ee
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"5639B67A2000","o":"230B5FE","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5639B67A2000","o":"230AEEF"},{"b":"5639B67A2000","o":"230AF8C"},{"b":"7F5F087D9000","o":"143C0"},{"b":"7F5F085E7000","o":"4303B","s":"gsignal"},{"b":"7F5F085E7000","o":"22859","s":"abort"},{"b":"5639B67A2000","o":"A76FDD","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5639B67A2000","o":"81EACE"},{"b":"5639B67A2000","o":"BB3429"},{"b":"5639B67A2000","o":"8290A8","s":"__wt_err"},{"b":"5639B67A2000","o":"829408","s":"__wt_panic"},{"b":"5639B67A2000","o":"8351D6"},{"b":"5639B67A2000","o":"C7C644","s":"__wt_block_extlist_read"},{"b":"5639B67A2000","o":"C7CB8C","s":"__wt_block_extlist_read_avail"},{"b":"5639B67A2000","o":"C79776","s":"__wt_block_checkpoint_load"},{"b":"5639B67A2000","o":"C5FE68"},{"b":"5639B67A2000","o":"BD81AB","s":"__wt_btree_open"},{"b":"5639B67A2000","o":"B59355","s":"__wt_conn_dhandle_open"},{"b":"5639B67A2000","o":"BB1CFE","s":"__wt_session_get_dhandle"},{"b":"5639B67A2000","o":"BB22F0","s":"__wt_session_get_dhandle"},{"b":"5639B67A2000","o":"BB24F4","s":"__wt_session_get_btree_ckpt"},{"b":"5639B67A2000","o":"C225B6","s":"__wt_curfile_open"},{"b":"5639B67A2000","o":"BADA3F"},{"b":"5639B67A2000","o":"B754E2","s":"__wt_metadata_cursor_open"},{"b":"5639B67A2000","o":"B755CF","s":"__wt_metadata_cursor"},{"b":"5639B67A2000","o":"B57FF7","s":"wiredtiger_open"},{"b":"5639B67A2000","o":"B2B438","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5639B67A2000","o":"B108F9"},{"b":"5639B67A2000","o":"CF4386","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5639B67A2000","o":"AEA659"},{"b":"5639B67A2000","o":"AEDB83","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"5639B67A2000","o":"A7B99D","s":"main"},{"b":"7F5F085E7000","o":"240B3","s":"__libc_start_main"},{"b":"5639B67A2000","o":"AD80EE","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.6.8", "gitVersion" : "8e540c0b6db93ce994cc548f000900bdc740f80a", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.15.0-69-generic", "version" : "#76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023", "machine" : "x86_64" }, "somap" : [ { "b" : "5639B67A2000", "elfType" : 3, "buildId" : "28879BCB86C4B9AF7CEC8E6482F331A095E338D0" }, { "b" : "7FFCA86F6000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "2DFFF2E3B9908E870457ADFD8AB8457ADCCEE5CA" }, { "b" : "7F5F09208000", "path" : "/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "F129B7D409F6F63A822673139BE40F369F3E4B13" }, { "b" : "7F5F091EC000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "7A04C793A2CE7B847F43840CA650E0D9DD35389D" }, { "b" : "7F5F091E1000", "path" : "/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "74CAFD320059B4D17CF89FD6379A5105FF55840F" }, { "b" : "7F5F09186000", "path" : "/lib/x86_64-linux-gnu/libyaml-cpp.so.0.6", "elfType" : 3, "buildId" : "B9EC1711A8AAD2FE980B735F13E186BEFE9FA4DB" }, { "b" : "7F5F0917B000", "path" : "/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "9199246C7429052FBE1F97DA6FE0B0CFBED2DE3F" }, { "b" : "7F5F090EC000", "path" : "/lib/x86_64-linux-gnu/libboost_program_options.so.1.71.0", "elfType" : 3, "buildId" : "CB2EB21D65AF49F187F384ABED5D46D66E998195" }, { "b" : "7F5F090CC000", "path" : "/lib/x86_64-linux-gnu/libboost_filesystem.so.1.71.0", "elfType" : 3, "buildId" : "956195B5ABA44A072A6242BF2399D9EB0B6163E7" }, { "b" : "7F5F08ED5000", "path" : "/lib/x86_64-linux-gnu/libtcmalloc.so.4", "elfType" : 3, "buildId" : "BC036BA2C0C0B4976A51184F1E9F712CCB8611C7" }, { "b" : "7F5F08EB9000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "5207766495EFFA5972E346F538623F8C00272A15" }, { "b" : "7F5F08BE3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "6BE4B3B97050EF9D14925AC22C4274C402AAD6D3" }, { "b" : "7F5F08B50000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "32C6D9DFC0E4AAD7CE47F7F6E63D44314FA9C1E1" }, { "b" : "7F5F08B4A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0CE76E566928EEB22CBB5282B50D7689DBAA076D" }, { "b" : "7F5F08966000", "path" : "/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "C90E6603C7CDF84713CD445700A575D3EA446D9B" }, { "b" : "7F5F08817000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D6BC45CB4B731E9BEA973FDFE568B431F2EACBF9" }, { "b" : "7F5F087FC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "4ABD133CC80E01BB388A9C42D9E3CB338836544A" }, { "b" : "7F5F087D9000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "F0983025F0E0F327A6DA752FF4FFA675E0BE393F" }, { "b" : "7F5F085E7000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "9FDB74E7B217D06C93172A8243F8547F947EE6D1" }, { "b" : "7F5F08574000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "AC83711FBCB2725AD376CA8F8C947F6A782E7A65" }, { "b" : "7F5F08555000", "path" : "/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "E4502D71B7D21D13E84436FAF0BD4A8FF63874C2" }, { "b" : "7F5F09261000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "39BD4A6852F6B9962AD57117D11B6F130D32F7FD" }, { "b" : "7F5F0852C000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "533783B82D9BEDA5891A5E10B15D43ADE31085CC" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x3E) [0x5639b8aad5fe]
 mongod(+0x230AEEF) [0x5639b8aaceef]
 mongod(+0x230AF8C) [0x5639b8aacf8c]
 libpthread.so.0(+0x143C0) [0x7f5f087ed3c0]
 libc.so.6(gsignal+0xCB) [0x7f5f0862a03b]
 libc.so.6(abort+0x12B) [0x7f5f08609859]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5639b7218fdd]
 mongod(+0x81EACE) [0x5639b6fc0ace]
 mongod(+0xBB3429) [0x5639b7355429]
 mongod(__wt_err+0xA5) [0x5639b6fcb0a8]
 mongod(__wt_panic+0x38) [0x5639b6fcb408]
 mongod(+0x8351D6) [0x5639b6fd71d6]
 mongod(__wt_block_extlist_read+0x94) [0x5639b741e644]
 mongod(__wt_block_extlist_read_avail+0x2C) [0x5639b741eb8c]
 mongod(__wt_block_checkpoint_load+0x226) [0x5639b741b776]
 mongod(+0xC5FE68) [0x5639b7401e68]
 mongod(__wt_btree_open+0xD4B) [0x5639b737a1ab]
 mongod(__wt_conn_dhandle_open+0x295) [0x5639b72fb355]
 mongod(__wt_session_get_dhandle+0xCE) [0x5639b7353cfe]
 mongod(__wt_session_get_dhandle+0x6C0) [0x5639b73542f0]
 mongod(__wt_session_get_btree_ckpt+0x154) [0x5639b73544f4]
 mongod(__wt_curfile_open+0x56) [0x5639b73c45b6]
 mongod(+0xBADA3F) [0x5639b734fa3f]
 mongod(__wt_metadata_cursor_open+0x72) [0x5639b73174e2]
 mongod(__wt_metadata_cursor+0x4F) [0x5639b73175cf]
 mongod(wiredtiger_open+0x2027) [0x5639b72f9ff7]
 mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xCE8) [0x5639b72cd438]
 mongod(+0xB108F9) [0x5639b72b28f9]
 mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E6) [0x5639b7496386]
 mongod(+0xAEA659) [0x5639b728c659]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x613) [0x5639b728fb83]
 mongod(main+0xD) [0x5639b721d99d]
 libc.so.6(__libc_start_main+0xF3) [0x7f5f0860b0b3]
 mongod(_start+0x2E) [0x5639b727a0ee]
-----  END BACKTRACE  -----
Aborted (core dumped) 

 

Comment by Chris Kelly [ 06/Jun/23 ]

This error message leads us to suspect some form of physical corruption. Please make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath.

The ideal resolution is to perform a clean resync from an unaffected node.

You can also try mongod --repair using the latest patch release of your version of MongoDB. Since you haven't specified what version you are using, I cannot advise the exact one and I'm not sure if you're using the latest patch release already.

In the event that a --repair operation is unsuccessful, then please also provide:

  • The logs leading up to the first occurrence of any issue
  • The logs of the repair operation.
  • The logs of any attempt to start mongod after the repair operation completed.

Christopher

 

Generated at Thu Feb 08 06:36:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.