[SERVER-30267] Mongod layer's hot backup Created: 21/Jul/17  Updated: 08/Oct/19  Resolved: 08/Oct/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: deyukong Assignee: Brian Lane
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to WT-3443 wt's backup method missed wiredtigerL... Closed
Participants:

 Description   

Hi, the physical backup strategy suggested by official mongo's document is to use
lvm, the filesystem level backup, which can be found here(https://docs.mongodb.com/manual/core/backups/).
It makes the process of initializing the machine more complex.
Physical backup is necessary because we can find a more convinent way to do initialSync,
No matter how fast the initialSync process can be after mongo 3.4, it can not be as fast as
physical backup.
I know there is a hotbackup api in wiredtiger layer(http://source.wiredtiger.com/2.5.0/backup.html), and percona also provides mongo layer's hotBackup command.
After some considerations, I find it not difficult to implement the command.
The best way is to forward the backup bytes onto a stream, the stream may be a tcp connection to the target machine, or a fd to the local disk. But mongo is only a service and does not have a sdk for users, so it may be considered futher.
What can be provided easily is what percona did, backuping up onto the local disk.

I think this improment is valuable and if you like, I'm glad to make this contribution.



 Comments   
Comment by Brian Lane [ 08/Oct/19 ]

Hi wolf_kdy,

We have completed this work as part of a separate project that got released in 4.2 enterprise server.

-Brian

Comment by deyukong [ 07/Dec/18 ]

@dmurphy 

Sorry for the delay.

1) This is a functionality, and certainly it most suits for initialSync, but in pure backup case, it also works

2) The security problem really matters, I agree

3) The bigest challange you mentioned is block-syncing files. I don't know why "block"-syncing is needed. Wt-layer has already guaranteed

that the partial read/write of a hot-backup files won't matter. You can refer to the manuals.

Comment by Jakub Kramarz [ 21/Aug/18 ]

Hi all,

From my point of view, the ideal solution would be something like they've implemented in Percona XtraBackup for MySQL Databases.

Aside from backing up to files, there's an option to stream backup (as a complete archive) to stdout without saving it do disk.
In this approach it is quite simple to use any backup system (eg. streaming plugin in Bareos).

In most basic implementation, it may be just a created on-a-fly tar archive, so no special tools would be needed for restore. 

I'd consider it as a must-have feature, because:

  • there's no need to have twice the space,
  • backups can be faster, because local disks are not busy writting backup,
  • one can layer own encryption and compression algorithms according to corporate requirements.

I'd rather provide additional utility to run localy on Mongo server (eg. connected via socket), instead of making mongod capable of external connections to make security simpler (no need to wait for remote connections, allow additional network traffic, nor change SELinux policies).

Regards,
Jakub

Comment by David Murphy [ 13/Mar/18 ]

@deyukong

There are a few things that seem unclear to me. When you suggest running "hot backup", you provide no output.

To me, this means you have a host we will call it node5, which is to replace a broken node. You would run the command from host 5, to clone another node's data into this new node. This is the reason your solution does not provide a destination when starting the hot backup. This also implied we must not allow MongoDB to pipe blocks over TCP which would indicate we are basically doing a form of ISCSI which makes we wonder about a good deal of issues. Furthermore, we would need to somehow delete the current dbpath, but allow MongoDB to still run so we could place the new files into this location.

As a result of all of this, it really sounds like your asking for a better initialSync maybe called "binarySync" which is not logical but binary in its nature? The biggest challenge I see in this approach is due to network time, getting a final block sync of all files could be very challenging, and we would have to expose the block level storage directly to an API interface, but I am sure security-minded folks would not prefer.

I would if a better option is to use the Percona like approach but to instead support tcp: //, tcps://, file:// type destinations, allowing streaming of a hot backup to avoid the disk burden you mention. Would be wrong in that view?

Comment by Ramon Fernandez Marina [ 08/Aug/17 ]

Thanks for the additional details wolf_kdy. This ticket has been place in the Storage team's backlog for future consideration.

Regards,
Ramón.

Comment by deyukong [ 21/Jul/17 ]

percona's hotbackup is good, but not enough.
It requires twice the original mongod's used space left on the disk, which is a strong restriction.
If we have a command on mongod, which may be like below
db.runCommand(

{hotbackup:1}

) replys with a token

{ok:1, token: "9131334421"}

db.runCommand(

{hotbackupContinue:1, token:"9131334421"}

replys each time with

{ok:1, filename:"WT_COLLECTION-13984883882", offset:123139943, complete:false,data:Binary("29481883ade1e1")}

)

{ok:1, filename:"WT_COLLECTION-13984883882", offset:123139950, complete:false,data:Binary("ada18384800388")}

)

{ok:1,complete:true}

the mongod layer keeps a backupContext, created when the hotbakup command is called, released when the backup procedure is finished, the backupContext holds the backup checkpoint of wiredTiger.

If we have this functionality , the hotbackup will be much better than the percona's. A more convinent alternative to initialSync.

Comment by deyukong [ 21/Jul/17 ]

Hi, Ramon, thanks for your reply.
1) how would one use it
Assume we have a repl with one primary and two secondaries, each on a physical machine. and now one machine crashed and can not be recovered soon. What we can do now is to add a new node and do initialSync, initialSync is logical syncing, records by records. index by index and very slow if dataset is large or users have too many indexes.
What we do in practical in fact is to make a physical copy from the living primary or secondary to the target new machine, restart and change the replset-view by db.reconfig(). It is not easy to get the db-level consistent files copied, unless the mongod on the machine is stopped or lvm snapshot is made. Neither of them are convinent.
Percona hot backup supplies another choice, we can make the physical files copy from the db-level mechanism, it is more convinent than lvm, more safe since there is no need to stop the remaining node.
2) how it works
In fact, I didnt understand what you really want to get from this question.
You may what to ask how percona's hot backup works.
Percona did not release their implementations, so I can not figure it out clearly, But I believe it uses wt's backup api to do this.
You may what to ask how wt's backup guarentees the copy is consistent even if reads and writes are both on the run.
Wt really guarentees this in fact, the meta data refers to different blocks of the file and if backup begins, we mvcc the metadata, does not change the old ones, certainly there is a chance that particial reads happen on the data blocks but it's just ok since the metadata does not refer to the particial written data blocks.

Comment by Ramon Fernandez Marina [ 21/Jul/17 ]

Hi wolf_kdy; I looked at the at the WT backups and the Percona hot backup, but I can't say I understand exactly what is the functionality you're proposing, how would it work, how would one use it, etc.

Will you please provide more details about what you have in mind?

Thanks,
Ramón.

Comment by deyukong [ 21/Jul/17 ]

lvm is too old, not convinent to use, and has a horrible performance in the stage of snapshot.
percona style's backup is more elegent but has the restriction of local disk.
Here a tool may be provided, a stub, which provides a tcp server, so users can choose to put the disk any where.
The tool may even be the mongod itself, which I think is the most elegent way.

Generated at Thu Feb 08 04:23:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.