[SERVER-39769] Empty WiredTiger.backup causes restore from backup to fail Created: 22/Feb/19  Updated: 01/Mar/19  Resolved: 01/Mar/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.2.21, 4.0.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Luke Jolly Assignee: Susan LoVerso
Resolution: Duplicate Votes: 1
Labels: SWNA
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates WT-4615 Sync backup file before returning bac... Closed
Operating System: ALL
Steps To Reproduce:
  1. Call db.fsyncLock()
  2. Take FS snapshot shortly after without syncing to disk
  3. Restore snapshot
  4. WiredTiger.backup is zero bytes
Sprint: Storage Engines 2019-03-11
Participants:

 Description   

After calling the db.fsyncLock() command and it returning, the WiredTiger.backup file is created. After waiting 20 seconds I make a Google Cloud disk snapshot. Restoring from this snapshot shows that WiredTiger.backup is zero bytes. This causes MongoDB to not be able to restore correctly on startup and there being no databases. Adding a manual call to the Linux sync command before making the snapshot remedies this. It seems like MongoDB should be syncing WiredTiger.backup to disk before the db.fsyncLock() command returns.

Ubuntu 18.04



 Comments   
Comment by Susan LoVerso [ 01/Mar/19 ]

You are correct, it does appear that we're not correctly syncing out the backup file. I have opened WT-4615 for the WiredTiger library changes to fix this issue.

Comment by Bruce Lucas (Inactive) [ 26/Feb/19 ]

I used strace to verify that we fsync neither the backup file nor the directory where it's located. Shouldn't we do that to ensure that fsyncLock persists everything required, in order to allow snapshot backups and also to protect against crashes while fsyncLock'ed?

Checked 4.0.6, reported above in 3.2, so I assume it affects all versions.

Comment by Luke Jolly [ 22/Feb/19 ]

When WiredTiger.backup is zero bytes and MongoDB fails to recover, this log happens:

"WiredTiger message Both WiredTiger.turtle and WiredTiger.backup exist; recreating metadata from backup"

Weirdly, WiredTiger.turtle and WiredTiger.backup always exist, but that log is only printed when WiredTiger.backup is zero bytes.

Generated at Thu Feb 08 04:53:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.