[SERVER-56695] WiredTiger.turtle: encountered an illegal file format or internal value Created: 06/May/21  Updated: 13/May/21  Resolved: 13/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.6.3
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Jack Roger Assignee: Edwin Zhou
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.tar.gz     Text File mongod-210508-111358.log     Text File repair-210507-101605.log     Text File repair-210508-111358.log     Text File repair.log     File repair.tar.gz    
Participants:

 Description   

Same issue as this one, the repair output is in the attachment, please help!



 Comments   
Comment by Edwin Zhou [ 13/May/21 ]

Hi lyzenghq@gmail.com,

Unfortunately we aren't able to repair this corruption.

To avoid a problem like this in the future, it is our strong recommendation to:

Best,
Edwin

Comment by Jack Roger [ 13/May/21 ]

Tried v3.6.23, still this error

 

2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] MongoDB starting : pid=92728 port=27017 dbpath=/home/user/mongodb/io01/var/lib/mongodb/ 64-bit host=mu01
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] db version v3.6.23
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] git version: d352e6a4764659e0d0350ce77279de3c1f243e5c
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] modules: none
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] build environment:
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten]     distmod: rhel70
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten]     distarch: x86_64
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2021-05-13T09:24:17.034+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/home/user/mongodb/io01/var/lib/mongodb/", engine: "wiredTiger" } }
2021-05-13T09:24:17.037+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2021-05-13T09:24:17.037+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=128194M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(
fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-05-13T09:24:17.645+0800 E STORAGE  [initandlisten] WiredTiger error (13) [1620869057:645700][92728:0x2b856e528d00], file:WiredTiger.wt, connection: __ckpt_version_chk, 596: file:WiredTiger.wt is an unsupported WiredTiger source file 
version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied Raw: [1620869057:645700][92728:0x2b856e528d00], file:WiredTiger.wt, connection: __ckpt_version_chk, 596: file:WiredTiger.wt is an unsupported Wir
edTiger source file version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied
2021-05-13T09:24:17.647+0800 E -        [initandlisten] Assertion: 28718:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 463
2021-05-13T09:24:17.647+0800 I STORAGE  [initandlisten] exception in initAndListen: Location28718: 13: Permission denied, terminating
2021-05-13T09:24:17.647+0800 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2021-05-13T09:24:17.647+0800 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2021-05-13T09:24:17.647+0800 I CONTROL  [initandlisten] now exiting
2021-05-13T09:24:17.647+0800 I CONTROL  [initandlisten] shutting down with code:100

Comment by Daniel Gottlieb (Inactive) [ 13/May/21 ]

Just a hunch lyzenghq@gmail.com, but can you try with a later version of 3.6? I see you're running on 3.6.3 and that error message looks like something I'd expect when trying to open datafiles that were output by a later version of 3.6.

Comment by Jack Roger [ 13/May/21 ]

I used v3.6 to run the repair and still have this error

 

2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] MongoDB starting : pid=20385 port=27017 dbpath=/var/lib/mongo/ 64-bit host=io01
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] db version v3.6.3
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] git version: nogitversion
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] modules: none
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] build environment:
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten]     distarch: x86_64
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2021-05-13T08:45:43.122+0800 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongo/", engine: "wiredTiger" } }
2021-05-13T08:45:43.125+0800 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
2021-05-13T08:45:43.125+0800 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=128282M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archi
ve=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2021-05-13T08:45:43.715+0800 E STORAGE  [initandlisten] WiredTiger error (13) [1620866743:715603][20385:0x7f4edae11b80], file:WiredTiger.wt, connection: file:WiredTiger.wt is an unsupported WiredTiger source file version 2.1; this WiredTi
ger build only supports versions from 1.1 to 1.1: Permission denied
2021-05-13T08:45:43.716+0800 E -        [initandlisten] Assertion: 28718:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 389
2021-05-13T08:45:43.716+0800 I STORAGE  [initandlisten] exception in initAndListen: Location28718: 13: Permission denied, terminating
2021-05-13T08:45:43.716+0800 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2021-05-13T08:45:43.716+0800 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2021-05-13T08:45:43.717+0800 I CONTROL  [initandlisten] now exiting
2021-05-13T08:45:43.717+0800 I CONTROL  [initandlisten] shutting down with code:100

Comment by Edwin Zhou [ 12/May/21 ]

Hi lyzenghq@gmail.com,

Can you attempt to run using MongoDB v3.6?

Best,
Edwin

Comment by Jack Roger [ 12/May/21 ]

And,

[root@io01 ~]# sudo -u mongodb mongod --version
db version v4.4.5
Build Info: {
    "version": "4.4.5",
    "gitVersion": "ff5cb77101b052fa02da43b8538093486cf9b3f7",
    "openSSLVersion": "OpenSSL 1.0.1e-fips 11 Feb 2013",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "rhel70",
        "distarch": "x86_64",
        "target_arch": "x86_64"
    }
}

Comment by Jack Roger [ 12/May/21 ]

The permissions of the files in $dbpath all seem to be correct, as detailed below.

 

[root@io01 ~]# ls -ld /var/lib/mongo/
drwxr-x--- 5 mongodb mongodb 32 May 12 08:42 /var/lib/mongo/
[root@io01 ~]#
[root@io01 ~]# ls -l /var/lib/mongo/
total 720464057
-rw------- 1 mongodb mongodb        16384 Apr 14 15:42 collection-0-3978976212545669616.wt
-rw------- 1 mongodb mongodb 328083353600 Apr 28 03:36 collection-0--4446081023038618115.wt
-rw------- 1 mongodb mongodb        36864 Apr 14 15:43 collection-2-3978976212545669616.wt
-rw------- 1 mongodb mongodb 170693124096 Apr 28 03:36 collection-3--4446081023038618115.wt
-rw------- 1 mongodb mongodb        36864 Apr 28 03:35 collection-4-3978976212545669616.wt
drwx------ 2 mongodb mongodb           23 Apr 28 03:36 diagnostic.data
-rw------- 1 mongodb mongodb  11099844608 Apr 28 03:36 index-10--4446081023038618115.wt
-rw------- 1 mongodb mongodb        16384 Apr 14 15:42 index-1-3978976212545669616.wt
-rw------- 1 mongodb mongodb  53919219712 Apr 28 03:36 index-1--4446081023038618115.wt
-rw------- 1 mongodb mongodb  33569361920 Apr 28 03:36 index-2--4446081023038618115.wt
-rw------- 1 mongodb mongodb        36864 Apr 14 15:43 index-3-3978976212545669616.wt
-rw------- 1 mongodb mongodb  43571269632 Apr 28 03:36 index-4--4446081023038618115.wt
-rw------- 1 mongodb mongodb        36864 Apr 26 11:15 index-5-3978976212545669616.wt
-rw------- 1 mongodb mongodb  26918404096 Apr 28 03:36 index-5--4446081023038618115.wt
-rw------- 1 mongodb mongodb        36864 Apr 28 03:35 index-6-3978976212545669616.wt
-rw------- 1 mongodb mongodb  26919858176 Apr 28 03:36 index-6--4446081023038618115.wt
-rw------- 1 mongodb mongodb  15400296448 Apr 28 03:36 index-7--4446081023038618115.wt
-rw------- 1 mongodb mongodb  16370061312 Apr 28 03:36 index-8--4446081023038618115.wt
-rw------- 1 mongodb mongodb  11209912320 Apr 28 03:36 index-9--4446081023038618115.wt
drwx------ 2 mongodb mongodb           22 May  7 10:16 journal
-rw------- 1 mongodb mongodb        36864 Apr 20 10:59 _mdb_catalog.wt
-rw------- 1 mongodb mongodb            0 May  6 14:20 mongod.lock
-rw------- 1 mongodb mongodb           73 May 12 08:42 _repair_incomplete
-rw------- 1 mongodb mongodb        36864 Apr 28 03:36 sizeStorer.wt
-rw------- 1 mongodb mongodb          114 Apr 14 15:17 storage.bson
drwx------ 2 mongodb mongodb            0 Apr 16 09:28 _tmp
-rw------- 1 mongodb mongodb           48 Apr 14 15:17 WiredTiger
-rw------- 1 mongodb mongodb         4096 Apr 14 15:42 WiredTigerLAS.wt
-rw------- 1 mongodb mongodb           21 Apr 14 15:17 WiredTiger.lock
-rw------- 1 mongodb mongodb         1452 May  7 23:49 WiredTiger.turtle
-rw------- 1 mongodb mongodb        94208 May  7 23:49 WiredTiger.wt
-rw------- 1 mongodb mongodb        94208 May 12 08:42 WiredTiger.wt.orig

Does the above error message mean that the version of the WiredTiger.wt file is not supported?

In addition, the repair command I used is:

sudo -u mongodb mongod --repair --dbpath /var/lib/mongo/ --storageEngine wiredTiger

Comment by Edwin Zhou [ 11/May/21 ]

There appears to be an issue with the permissions of the files in the repair attempt I provided based on these log lines:

{"t":{"$date":"2021-05-08T11:13:58.974+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1620443638:974462][59086:0x7f77817f8bc0], file:WiredTiger.wt, connection: __ckpt_version_chk, 1112: file:WiredTiger.wt is an unsupported WiredTiger source file version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied"}}
{"t":{"$date":"2021-05-08T11:13:58.985+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1620443638:985351][59086:0x7f77817f8bc0], file:WiredTiger.wt, connection: __ckpt_version_chk, 1112: file:WiredTiger.wt is an unsupported WiredTiger source file version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied"}}
{"t":{"$date":"2021-05-08T11:13:58.996+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1620443638:996098][59086:0x7f77817f8bc0], file:WiredTiger.wt, connection: __ckpt_version_chk, 1112: file:WiredTiger.wt is an unsupported WiredTiger source file version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied"}}
{"t":{"$date":"2021-05-08T11:13:58.997+08:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2021-05-08T11:13:58.997+08:00"},"s":"W",  "c":"STORAGE",  "id":22349,   "ctx":"initandlisten","msg":"Attempting to salvage WiredTiger metadata"}
{"t":{"$date":"2021-05-08T11:13:59.119+08:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1620443639:119295][59086:0x7f77817f8bc0], file:WiredTiger.wt, WT_SESSION.salvage: __ckpt_version_chk, 1112: file:WiredTiger.wt is an unsupported WiredTiger source file version 2.1; this WiredTiger build only supports versions from 1.1 to 1.1: Permission denied"}}
{"t":{"$date":"2021-05-08T11:13:59.120+08:00"},"s":"F",  "c":"STORAGE",  "id":50947,   "ctx":"initandlisten","msg":"Failed to salvage WiredTiger metadata","attr":{"details":"13: Permission denied"}}

Can you please verify that the permissions of the repair files are correctly set?

Best,
Edwin

Comment by Jack Roger [ 08/May/21 ]

After replacing the files in tar, the number of errors is reduced but still exists. The repair output is in the attachment repair-210508-111358.log, and the service log is in mongod-210508-111358.log

Comment by Edwin Zhou [ 07/May/21 ]

Hi lyzenghq@gmail.com,

I've attached a repair attempt of the files you provided as repair.tar.gz. Please extract these files, replace them in your $dbpath, and let us know if it resolves the issue.

Best,
Edwin

Comment by Jack Roger [ 07/May/21 ]

Use mongodb 4.4 to repair, still failed. The repair output is in the attachment repair-210507-101605.log

Comment by Edwin Zhou [ 06/May/21 ]

Hi lyzenghq@gmail.com,
MongoDB 3.6 reached end of life in April of 2021. But we can provide limited guidance on this issue.

First, make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath. Then, try mongod --repair using the latest version of MongoDB.

Best,
Edwin

Generated at Thu Feb 08 05:39:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.