[SERVER-30267] Mongod layer's hot backup Created: 21/Jul/17 Updated: 08/Oct/19 Resolved: 08/Oct/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | deyukong | Assignee: | Brian Lane |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
Hi, the physical backup strategy suggested by official mongo's document is to use I think this improment is valuable and if you like, I'm glad to make this contribution. |
| Comments |
| Comment by Brian Lane [ 08/Oct/19 ] |
|
Hi wolf_kdy, We have completed this work as part of a separate project that got released in 4.2 enterprise server. -Brian |
| Comment by deyukong [ 07/Dec/18 ] |
|
Sorry for the delay. 1) This is a functionality, and certainly it most suits for initialSync, but in pure backup case, it also works 2) The security problem really matters, I agree 3) The bigest challange you mentioned is block-syncing files. I don't know why "block"-syncing is needed. Wt-layer has already guaranteed that the partial read/write of a hot-backup files won't matter. You can refer to the manuals. |
| Comment by Jakub Kramarz [ 21/Aug/18 ] |
|
Hi all, From my point of view, the ideal solution would be something like they've implemented in Percona XtraBackup for MySQL Databases. Aside from backing up to files, there's an option to stream backup (as a complete archive) to stdout without saving it do disk. In most basic implementation, it may be just a created on-a-fly tar archive, so no special tools would be needed for restore. I'd consider it as a must-have feature, because:
I'd rather provide additional utility to run localy on Mongo server (eg. connected via socket), instead of making mongod capable of external connections to make security simpler (no need to wait for remote connections, allow additional network traffic, nor change SELinux policies). Regards, |
| Comment by David Murphy [ 13/Mar/18 ] |
|
@deyukong There are a few things that seem unclear to me. When you suggest running "hot backup", you provide no output. To me, this means you have a host we will call it node5, which is to replace a broken node. You would run the command from host 5, to clone another node's data into this new node. This is the reason your solution does not provide a destination when starting the hot backup. This also implied we must not allow MongoDB to pipe blocks over TCP which would indicate we are basically doing a form of ISCSI which makes we wonder about a good deal of issues. Furthermore, we would need to somehow delete the current dbpath, but allow MongoDB to still run so we could place the new files into this location. As a result of all of this, it really sounds like your asking for a better initialSync maybe called "binarySync" which is not logical but binary in its nature? The biggest challenge I see in this approach is due to network time, getting a final block sync of all files could be very challenging, and we would have to expose the block level storage directly to an API interface, but I am sure security-minded folks would not prefer. I would if a better option is to use the Percona like approach but to instead support tcp: //, tcps://, file:// type destinations, allowing streaming of a hot backup to avoid the disk burden you mention. Would be wrong in that view? |
| Comment by Ramon Fernandez Marina [ 08/Aug/17 ] |
|
Thanks for the additional details wolf_kdy. This ticket has been place in the Storage team's backlog for future consideration. Regards, |
| Comment by deyukong [ 21/Jul/17 ] |
|
percona's hotbackup is good, but not enough. ) replys with a token {ok:1, token: "9131334421"}db.runCommand( {hotbackupContinue:1, token:"9131334421"}replys each time with {ok:1, filename:"WT_COLLECTION-13984883882", offset:123139943, complete:false,data:Binary("29481883ade1e1")}) {ok:1, filename:"WT_COLLECTION-13984883882", offset:123139950, complete:false,data:Binary("ada18384800388")}) {ok:1,complete:true}the mongod layer keeps a backupContext, created when the hotbakup command is called, released when the backup procedure is finished, the backupContext holds the backup checkpoint of wiredTiger. If we have this functionality , the hotbackup will be much better than the percona's. A more convinent alternative to initialSync. |
| Comment by deyukong [ 21/Jul/17 ] |
|
Hi, Ramon, thanks for your reply. |
| Comment by Ramon Fernandez Marina [ 21/Jul/17 ] |
|
Hi wolf_kdy; I looked at the at the WT backups and the Percona hot backup, but I can't say I understand exactly what is the functionality you're proposing, how would it work, how would one use it, etc. Will you please provide more details about what you have in mind? Thanks, |
| Comment by deyukong [ 21/Jul/17 ] |
|
lvm is too old, not convinent to use, and has a horrible performance in the stage of snapshot. |