[SERVER-39769] Empty WiredTiger.backup causes restore from backup to fail Created: 22/Feb/19 Updated: 01/Mar/19 Resolved: 01/Mar/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.21, 4.0.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Luke Jolly | Assignee: | Susan LoVerso |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | SWNA | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: |
|
||||||||
| Sprint: | Storage Engines 2019-03-11 | ||||||||
| Participants: | |||||||||
| Description |
|
After calling the db.fsyncLock() command and it returning, the WiredTiger.backup file is created. After waiting 20 seconds I make a Google Cloud disk snapshot. Restoring from this snapshot shows that WiredTiger.backup is zero bytes. This causes MongoDB to not be able to restore correctly on startup and there being no databases. Adding a manual call to the Linux sync command before making the snapshot remedies this. It seems like MongoDB should be syncing WiredTiger.backup to disk before the db.fsyncLock() command returns. Ubuntu 18.04 |
| Comments |
| Comment by Susan LoVerso [ 01/Mar/19 ] |
|
You are correct, it does appear that we're not correctly syncing out the backup file. I have opened |
| Comment by Bruce Lucas (Inactive) [ 26/Feb/19 ] |
|
I used strace to verify that we fsync neither the backup file nor the directory where it's located. Shouldn't we do that to ensure that fsyncLock persists everything required, in order to allow snapshot backups and also to protect against crashes while fsyncLock'ed? Checked 4.0.6, reported above in 3.2, so I assume it affects all versions. |
| Comment by Luke Jolly [ 22/Feb/19 ] |
|
When WiredTiger.backup is zero bytes and MongoDB fails to recover, this log happens: "WiredTiger message Both WiredTiger.turtle and WiredTiger.backup exist; recreating metadata from backup" Weirdly, WiredTiger.turtle and WiredTiger.backup always exist, but that log is only printed when WiredTiger.backup is zero bytes. |