[SERVER-40214] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 19/Mar/19  Updated: 22/Mar/19  Resolved: 22/Mar/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.4.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alexandre Baril-Lagace Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: wt-repair-success
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive WiredTiger.zip    
Operating System: ALL
Participants:

 Description   

Hello,

 

After a bad/crash stop of the mongo instance i can not restart i du to a corupted recovery file i think

I have seen other post where it seems possible for you to repair le file and get it back

I am currently down on a graylog server using mongo i woul be so greatfull if you could help!

 

I have attache the 2 WiredTiger files that seem necessary for this

 

And her are the logs i have on startup

 

some-mongo_1 | 2019-03-19T15:31:41.376+0000 W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
some-mongo_1 | 2019-03-19T15:31:41.420+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
some-mongo_1 | 2019-03-19T15:31:41.420+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
some-mongo_1 | 2019-03-19T15:31:41.420+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1335M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
some-mongo_1 | 2019-03-19T15:31:41.429+0000 E STORAGE [initandlisten] WiredTiger error (0) [1553009501:429266][1:0x7f999b7a2cc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
some-mongo_1 | 2019-03-19T15:31:41.429+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1553009501:429343][1:0x7f999b7a2cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
some-mongo_1 | 2019-03-19T15:31:41.429+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
some-mongo_1 | 2019-03-19T15:31:41.429+0000 I - [initandlisten]
some-mongo_1 |
some-mongo_1 | ***aborting after fassert() failure
some-mongo_1 |
some-mongo_1 |
some-mongo_1 | 2019-03-19T15:31:41.463+0000 F - [initandlisten] Got signal: 6 (Aborted).



 Comments   
Comment by Danny Hatcher (Inactive) [ 22/Mar/19 ]

Hello Alexandre,

I'm glad to hear it! We made some large improvements to our repair functionality in MongoDB 4.0 so please try that process going forward if you run into more issues. We also strongly recommend implementing a replica set across different servers as that will allow you to keep operating if one server goes down.

Thanks,

Danny

Comment by Alexandre Baril-Lagace [ 21/Mar/19 ]

Hello Dany,

That did the trick!

I ran

docker run --rm -v [local path to data]:/data/db --name mongo_fix mongo:4 mongod --dbpath /data/db --repair

then 

docker run -d -v [local path to data]:/data/db --name mongo_fix mongo:3

And i was able to access de data

It works and i learned something. Thank you very much! 

Comment by Danny Hatcher (Inactive) [ 21/Mar/19 ]

Hello Alexandre,

Please do the following:

1. Ensure you maintain a backup of those files.
2. Download the 4.0.6 binaries that match your operating system.
3. Using the 4.0.6 binary, run mongod --repair using the same dbpath as your files.
4. Using your original binary, run mongod normally.

Does that successfully allow you to start up?

Thank you,

Danny

Comment by Alexandre Baril-Lagace [ 20/Mar/19 ]

Hello Danny,
 
This mongoDB was running as a docker container in a Graylog stack/service
The mongo container has been deleted and recreated while attempting to recover the service
I only have the files from the bind mount pointing to the this path inside the mongo container —> /data/db
 
Here is what i have left from that directory
I don’t know if any of these files can help
 
I know the host has ran out of disk space an I suspect an unwritten or badly terminated files du to that 
 
rw-rr- 1 root root  45056 Mar 19 12:05 ./collection-0-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-0-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-0--6761012070591444171.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-0--699962151772404175.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-0-778621662237703678.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-12-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-1--4274925988054194855.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-15-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-18-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-21-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./collection-2-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-24-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-2--6761012070591444171.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./collection-2--699962151772404175.wt
rw-rr- 1 root root  24576 Mar 19 12:05 ./collection-27-3973860549754299777.wt
rw-rr- 1 root root 159744 Mar 19 12:05 ./collection-30-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-34-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-3-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-37-3973860549754299777.wt
rw-rr- 1 root root  20480 Mar 19 12:05 ./collection-39-3973860549754299777.wt
rw-rr- 1 root root  20480 Mar 19 12:05 ./collection-41-3973860549754299777.wt
rw-rr- 1 root root  77824 Mar 19 12:05 ./collection-43-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-4--4274925988054194855.wt
rw-rr- 1 root root  24576 Mar 19 12:05 ./collection-45-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-4--6761012070591444171.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-47-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-49-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-5-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./collection-6-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-7--4274925988054194855.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./collection-9-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-0--4274925988054194855.wt
rw-rr- 1 root root  32768 Mar 19 12:05 ./index-10-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-10--4274925988054194855.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-10-5995923855104597307.wt
rw-rr- 1 root root  32768 Mar 19 12:05 ./index-11-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-11--4274925988054194855.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-12--4274925988054194855.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-13-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-1-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-14-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-1-5995923855104597307.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-16-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-1--6761012070591444171.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-1--699962151772404175.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-17-3973860549754299777.wt
rw-rr- 1 root root  32768 Mar 19 12:05 ./index-1-778621662237703678.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-19-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-20-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-22-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-23-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-2--4274925988054194855.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-25-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-2-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-26-3973860549754299777.wt
rw-rr- 1 root root  24576 Mar 19 12:05 ./index-28-3973860549754299777.wt
rw-rr- 1 root root  24576 Mar 19 12:05 ./index-29-3973860549754299777.wt
rw-rr- 1 root root  57344 Mar 19 12:05 ./index-31-3973860549754299777.wt
rw-rr- 1 root root  53248 Mar 19 12:05 ./index-32-3973860549754299777.wt
rw-rr- 1 root root  98304 Mar 19 12:05 ./index-33-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-3-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-3--4274925988054194855.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-35-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-36-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-3--6761012070591444171.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-3--699962151772404175.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-38-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-40-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-42-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-4-3973860549754299777.wt
rw-rr- 1 root root  53248 Mar 19 12:05 ./index-44-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-4-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-46-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-48-3973860549754299777.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-50-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-5--4274925988054194855.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-5-5995923855104597307.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-5--6761012070591444171.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-6-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-6--4274925988054194855.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-7-3973860549754299777.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-7-5995923855104597307.wt
rw-rr- 1 root root  36864 Mar 19 12:05 ./index-8-3973860549754299777.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-8--4274925988054194855.wt
rw-rr- 1 root root   4096 Mar 19 12:05 ./index-8-5995923855104597307.wt
rw-rr- 1 root root  16384 Mar 19 12:05 ./index-9--4274925988054194855.wt
rw-rr- 1 root root  45056 Mar 19 12:05 ./index-9-5995923855104597307.wt
rw-rr- 1 root root  40960 Mar 19 12:05 ./_mdb_catalog.wt
rw-rr- 1 root root      2 Mar 19 12:05 ./mongod.lock
rw-rr- 1 root root  36864 Mar 19 12:05 ./sizeStorer.wt
rw-rr- 1 root root     95 Mar 19 12:05 ./storage.bson
rw-rr- 1 root root     46 Mar 19 12:05 ./WiredTiger
rw-rr- 1 root root   4096 Mar 19 12:05 ./WiredTigerLAS.wt
rw-rr- 1 root root     21 Mar 19 12:05 ./WiredTiger.lock
rw-rr- 1 root root    922 Mar 19 12:05 ./WiredTiger.turtle
-rwxr-xr-x 1 root root 380928 Mar 19 12:05 ./WiredTiger.wt
 
./diagnostic.data:
total 197980
drwxr-xr-x 2 root root     4096 Mar 19 12:05 .
drwxr-xr-x 4 root root     8192 Mar 19 12:05 ..
rw-rr- 1 root root 10485278 Mar 19 12:05 metrics.2019-02-06T17-49-05Z-00000
rw-rr- 1 root root 10484375 Mar 19 12:05 metrics.2019-02-08T16-04-05Z-00000
rw-rr- 1 root root 10484924 Mar 19 12:05 metrics.2019-02-10T21-04-06Z-00000
rw-rr- 1 root root 10479155 Mar 19 12:05 metrics.2019-02-12T23-29-09Z-00000
rw-rr- 1 root root  3175723 Mar 19 12:05 metrics.2019-02-15T04-57-05Z-00000
rw-rr- 1 root root    28979 Mar 19 12:05 metrics.2019-02-15T20-11-57Z-00000
rw-rr- 1 root root 10472778 Mar 19 12:05 metrics.2019-02-15T20-14-13Z-00000
rw-rr- 1 root root 10478177 Mar 19 12:05 metrics.2019-02-18T02-52-44Z-00000
rw-rr- 1 root root 10478753 Mar 19 12:05 metrics.2019-02-20T05-52-29Z-00000
rw-rr- 1 root root 10479657 Mar 19 12:05 metrics.2019-02-22T08-07-30Z-00000
rw-rr- 1 root root 10481572 Mar 19 12:05 metrics.2019-02-24T12-17-30Z-00000
rw-rr- 1 root root 10481399 Mar 19 12:05 metrics.2019-02-26T15-47-31Z-00000
rw-rr- 1 root root 10480813 Mar 19 12:05 metrics.2019-02-28T15-12-31Z-00000
rw-rr- 1 root root 10483927 Mar 19 12:05 metrics.2019-03-02T18-22-31Z-00000
rw-rr- 1 root root 10481967 Mar 19 12:05 metrics.2019-03-04T23-09-02Z-00000
rw-rr- 1 root root 10470919 Mar 19 12:05 metrics.2019-03-07T01-19-03Z-00000
rw-rr- 1 root root 10473949 Mar 19 12:05 metrics.2019-03-09T03-44-06Z-00000
rw-rr- 1 root root 10476608 Mar 19 12:05 metrics.2019-03-11T09-59-06Z-00000
rw-rr- 1 root root 10481320 Mar 19 12:05 metrics.2019-03-13T07-44-08Z-00000
rw-rr- 1 root root 10472472 Mar 19 12:05 metrics.2019-03-15T08-29-09Z-00000
rw-rr- 1 root root 10481899 Mar 19 12:05 metrics.2019-03-17T11-19-10Z-00000
rw-rr- 1 root root   339037 Mar 19 12:05 metrics.2019-03-19T12-39-11Z-00000
rw-rr- 1 root root    16743 Mar 19 12:05 metrics.interim
 
./journal:
total 307212
drwxr-xr-x 2 root root       110 Mar 19 12:05 .
drwxr-xr-x 4 root root      8192 Mar 19 12:05 ..
rw-rr- 1 root root 104857600 Mar 19 12:05 WiredTigerLog.0000000214
rw-rr- 1 root root 104857600 Mar 19 12:05 WiredTigerPreplog.0000000007
rw-rr- 1 root root 104857600 Mar 19 12:05 WiredTigerPreplog.0000000009

Comment by Danny Hatcher (Inactive) [ 20/Mar/19 ]

Hello Alexandre,

Could you please attach the mongod.log covering the time period from before the shutdown so that I can determine the source of the error?

Thank you,

Danny

Generated at Thu Feb 08 04:54:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.