[SERVER-39266] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 29/Jan/19  Updated: 20/Feb/19  Resolved: 31/Jan/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.16
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dmitrii Zotov Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu


Attachments: Zip Archive WiredTiger.zip     File new_repair.tar     File repair.tar    
Issue Links:
Related
related to WT-4554 Enhance WT salvage to handle the case... Closed
Operating System: ALL
Participants:

 Description   

Hi!

There was an unexpectedly reboot on the server. And I don't have backups.

Tried to upgrade from 3.4.16 to 4.0.5 and start with --repair option, but it doesn't help.

Thanks in advance!

Log file output:

2019-01-29T19:35:17.988+0200 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] MongoDB starting : pid=3552 port=27017 dbpath=/var/lib/mongodb 64-bit host=locean
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] db version v4.0.5
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] git version: 3739429dd92b92d1b0ab120911a23d50bf03c412
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] modules: none
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] build environment:
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten]     distarch: x86_64
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2019-01-29T19:35:17.994+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, repair: true$
2019-01-29T19:35:18.006+0200 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-01-29T19:35:18.006+0200 I STORAGE  [initandlisten]
2019-01-29T19:35:18.006+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-01-29T19:35:18.006+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-01-29T19:35:18.006+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=t$
2019-01-29T19:35:18.545+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548783318:545023][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-29T19:35:18.545+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548783318:545064][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-29T19:35:18.548+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548783318:548823][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-29T19:35:18.548+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548783318:548860][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-29T19:35:18.552+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548783318:552549][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-29T19:35:18.552+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548783318:552619][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-29T19:35:18.553+0200 W STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2019-01-29T19:35:18.553+0200 F STORAGE  [initandlisten] Reason: -31804: WT_PANIC: WiredTiger library panic
2019-01-29T19:35:18.553+0200 W STORAGE  [initandlisten] Attempting to salvage WiredTiger metadata
2019-01-29T19:35:18.573+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548783318:573691][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, WT_SESSION.salvage: __wt_turtle_read, 336: WiredTiger.turtle: fata$
2019-01-29T19:35:18.573+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548783318:573750][3552:0x7fd4dc5b1a40], file:WiredTiger.wt, WT_SESSION.salvage: __wt_panic, 523: the process must exit and res$
2019-01-29T19:35:18.574+0200 F STORAGE  [initandlisten] Failed to salvage WiredTiger metadata: -31804: WT_PANIC: WiredTiger library panic
2019-01-29T19:35:18.574+0200 F -        [initandlisten] Fatal Assertion 50947 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 653
2019-01-29T19:35:18.574+0200 F -        [initandlisten]***aborting after fassert() failure



 Comments   
Comment by Louis Williams [ 20/Feb/19 ]

rafald121 you are hitting the bug in SERVER-38177. You can work around this issue by not using the -bind_ip option with -repair, or upgrade to at least version 4.0.6.

In the future, please open new tickets rather than commenting on closed issues like this one. It is harder for us to track, and may result in a slower response for you.

Comment by Rafał Dołęga [ 19/Feb/19 ]

@Daniel Hatcher in my case one of collection has 15milions of rows and I am running this repair in Docker container. After trying to repair databases and this collection I got errors

 

mongo1_1         | 2019-02-19T20:11:28.576+0000 I STORAGE  [initandlisten] Repairing collection test.historic_row
mongo1_1         | 2019-02-19T20:11:28.578+0000 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-2040-6463085938029679551. Not salvaging.
mongo1_1         | 2019-02-19T20:11:28.590+0000 I INDEX    [initandlisten] build index on: test.historic_row properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "test.historic_row" }
mongo1_1         | 2019-02-19T20:11:28.590+0000 I INDEX    [initandlisten]       building index using bulk method; build may temporarily use up to 250 megabytes of RAM
mongo1_1         | 2019-02-19T20:11:28.600+0000 I INDEX    [initandlisten] build index on: test.historic_row properties: { v: 2, unique: true, key: { pair: 1, date: 1 }, name: "pair_1_date_1", ns: "test.historic_row", background: false, sparse: false }
mongo1_1         | 2019-02-19T20:11:28.600+0000 I INDEX    [initandlisten]       building index using bulk method; build may temporarily use up to 250 megabytes of RAM
mongo1_1         | 2019-02-19T20:11:28.624+0000 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
mongo1_1         | 2019-02-19T20:11:28.637+0000 F -        [initandlisten] Invalid access at address: 0
mongo1_1         | 2019-02-19T20:11:28.658+0000 F -        [initandlisten] Got signal: 11 (Segmentation fault).

 

 

Have you got any recommendation for me ? 

Comment by Danny Hatcher (Inactive) [ 31/Jan/19 ]

Hello Dmitrii,

Thank you for the update!

Have a great day,

Danny

Comment by Dmitrii Zotov [ 31/Jan/19 ]

Hello Daniel,

Thanks a lot! It's fully repaired. I appreciate your support.

Comment by Danny Hatcher (Inactive) [ 30/Jan/19 ]

Hello Dmitrii,

Can you try using this new new_repair.tar instead?

Thank you,

Danny

Comment by Dmitrii Zotov [ 30/Jan/19 ]

Thanks. Here is the log:

2019-01-30T06:25:52.512+0200 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] MongoDB starting : pid=15585 port=27017 dbpath=/var/lib/mongodb 64-bit host=locean
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] db version v4.0.5
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] git version: 3739429dd92b92d1b0ab120911a23d50bf03c412
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] modules: none
2019-01-30T06:25:52.518+0200 I CONTROL  [initandlisten] build environment:
2019-01-30T06:25:52.519+0200 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-01-30T06:25:52.519+0200 I CONTROL  [initandlisten]     distarch: x86_64
2019-01-30T06:25:52.519+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2019-01-30T06:25:52.519+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, repair: true$
2019-01-30T06:25:52.527+0200 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-01-30T06:25:52.527+0200 I STORAGE  [initandlisten]
2019-01-30T06:25:52.527+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-01-30T06:25:52.527+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-01-30T06:25:52.527+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=t$
2019-01-30T06:25:53.061+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548822353:61198][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-30T06:25:53.061+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548822353:61233][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-30T06:25:53.065+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548822353:65085][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-30T06:25:53.065+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548822353:65135][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-30T06:25:53.068+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548822353:68612][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_turtle_read, 336: WiredTiger.turtle: fatal turtle$
2019-01-30T06:25:53.068+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548822353:68652][15585:0x7f32226b2a40], file:WiredTiger.wt, connection: __wt_panic, 523: the process must exit and restart: WT$
2019-01-30T06:25:53.069+0200 W STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
2019-01-30T06:25:53.069+0200 F STORAGE  [initandlisten] Reason: -31804: WT_PANIC: WiredTiger library panic
2019-01-30T06:25:53.069+0200 W STORAGE  [initandlisten] Attempting to salvage WiredTiger metadata
2019-01-30T06:25:53.088+0200 E STORAGE  [initandlisten] WiredTiger error (-31803) [1548822353:88128][15585:0x7f32226b2a40], file:WiredTiger.wt, WT_SESSION.salvage: __wt_turtle_read, 336: WiredTiger.turtle: fata$
2019-01-30T06:25:53.088+0200 E STORAGE  [initandlisten] WiredTiger error (-31804) [1548822353:88170][15585:0x7f32226b2a40], file:WiredTiger.wt, WT_SESSION.salvage: __wt_panic, 523: the process must exit and res$
2019-01-30T06:25:53.089+0200 F STORAGE  [initandlisten] Failed to salvage WiredTiger metadata: -31804: WT_PANIC: WiredTiger library panic
2019-01-30T06:25:53.089+0200 F -        [initandlisten] Fatal Assertion 50947 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 653
2019-01-30T06:25:53.089+0200 F -        [initandlisten]***aborting after fassert() failure

Comment by Danny Hatcher (Inactive) [ 29/Jan/19 ]

Hello Dmitrii,

Thank you for your report. I've attached a repair attempt repair.tar of the files you provided. Please extract these files and replace them in your $dbpath and let us know if it resolves the issue.

Thank you,

Danny

Generated at Thu Feb 08 04:51:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.