[SERVER-23573] Backup/Restore of individual databases/collections with FS backups Created: 06/Apr/16  Updated: 07/Apr/23  Resolved: 08/Apr/22

Status: Closed
Project: Core Server
Component/s: Usability, WiredTiger
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Alexander Komyagin Assignee: Backlog - Storage Execution Team
Resolution: Done Votes: 11
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-29559 Don't crash the server when a databas... Closed
is duplicated by SERVER-29558 Keep all database state inside the da... Closed
Related
related to SERVER-19043 Implement a connectDatabase / disconn... Closed
is related to SERVER-29557 Allow healthy databases to skip repairs Closed
Assigned Teams:
Storage Execution
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2021-01-11, Execution Team 2021-01-25, Execution Team 2021-02-08, Execution Team 2021-02-22, Execution Team 2021-03-08, Execution Team 2021-03-22, Execution Team 2021-04-05, Execution Team 2021-04-19
Participants:
Case:

 Description   

For big deployments where mongodump/mongorestore approach is not feasible and backups are taken at FS level, we need way to restore individual databases and/or collections to the original system and to some other, new system. This feature is critical for success of those users who have multi-TB multi-tenant clusters and only want to restore a part of it.

With MMAP you could do the per-db backup/restore when directoryPerDb is used, but we no longer have the ability to use that approach in WT as WT metadata files are global for all namespaces. It will segfault if some *.wt files are not present.

However, with WT we have the advantage of individual files per namespace, so it should be possible to extract them and their metadata from an existing dbpath. Perhaps we will need a separate tool for that.



 Comments   
Comment by Rodrigo Nascimento [ 08/Apr/22 ]

Any chances you can review the decision of making this feature available only on Enterprise Advanced? We are talking about a feature that was available in Community via MMAPv1 and went away when MMAPv1 got deprecated. So, why not give it back to Community? 

Comment by Evin Roesle [ 08/Apr/22 ]

This work is completed and it will be available in future versions of Enterprise Advanced. As previously mentioned, this will not be available in future Community versions due to implementation requirements that are not supported in Community.

Comment by Paul Reed [ 19/Apr/21 ]

Thats a shame - only today I had an issue where some data had been deleted in a membership database. So rather than restore from a lengthy back up process, I simply copied a backup folder of the database in question to another directory. Pointed a fresh instance of mongod at it, and was able to immediately pull the deleted data and resurrect it in the other live running instance. Having folder connectivity for databases is so so so effective and time saving. Please please please find the resource to bring this functionality to WiredTiger.

Comment by Michael Gargiulo [ 19/Apr/21 ]

paul.reed@gprxdata.com This is not actively being worked on. Per my last comment, we have internal SaaS product features coming out in the next major release of MongoDB that address this request in some capacity, but currently do not have a plan to release this in Community. We are exploring additional projects that will accomplish this ask and are defining requirements with this request in mind, but have no current timetable for when active development will begin or features will be released to Community. 

I am putting this ticket into our backlog until we actively begin development or plan to release the upcoming features mentioned above on Community. 

Comment by Paul Reed [ 19/Apr/21 ]

Is this just being kicked down the road in sprints or is it actively being worked on. 

Comment by Paul Reed [ 05/Jan/21 ]

Please release this in community version.

Comment by Michael Gargiulo [ 05/Jan/21 ]

paul.reed@gprxdata.com This feature, in some capacity, full import/export of live collection files, will be available in the next major release of MongoDB in 2021.

However, at this time, this feature will not be generally available in Community or Enterprise Advanced, but will only be available for internal use in our SaaS offering. 

We are exploring other work that will have selective import/export capabilities, as fully requested in this ticket, along with how to release this for general use. 

Comment by Paul Reed [ 10/Dec/20 ]

Hoooooorrrraaaayyyyyyyyyyyy

Finally I will be able to move to WT and refactor out all the old work arounds. I am sooooo happy about this, only took 5 years !!!

Any insight into how this will materialize itself ?

 

Comment by Rodrigo Nascimento [ 08/Oct/19 ]

I share your pain Paul Reed. Looking forward to seeing this feature implemented within Wiredtiger. 

I understand it would involve some changes in the way WiredTiger deals with metadata, but from my perspective, it could be done.

Comment by Paul Reed [ 02/Oct/19 ]

Another year goes by, and another time I plead for this ticket to be given some airtime.

Is there any movement forward with a solution which would work, and thus allow me to move to the latest version.

 

Comment by Paul Reed [ 04/Jul/18 ]

Now MMAPv is being deprecated.

Can we have this feature please or SERVER_1903. I would truly love to move to WiredTiger, but cannot without folder level backup/restores.

Also - I worry about how to restore in the case of a catalog/meta file corruption - which I have seen several posts about with complicated salvage operations required to bring even a small database back in.

 

Please please please prioritize this.

 

 

 

Comment by Chris Kuethe [ 16/Jun/17 ]

I would very much like to see this happen.

Comment by Konstantin Manchev [ 26/Nov/16 ]

We face the same issue as well , we have 4TB databases in WT(snappy) same cluster , and there is multiple changes in one of them which is ~500GB , with the mmapv1 it was easy just to backup/restore the file system folder snapshot inside dbPath , now with WT we need to backup/restore full fs snapshot of all the 4TB considering the rest of databases are not changing often.

Moreover using mongodump/mongorestore on huge snappy/zlib WT compressed databases it just takes too much time as the mongodump is uncompressed and sending/receiving data to the mongod is just times slower than filesystem snapshot ...

I guess splitting the metadata in dbPath in separate files for every database folder inside dbPath shall not be such difficult task for the dev team considering the database wt-files are already separated in the folders?

Cheers,
Kiko

Comment by Rodrigo Nascimento [ 11/Oct/16 ]

I just got trapped by this issue.

I was trying to implement a backup/restore strategy that requires a per-db restore, but because of WiredTiger global metadata I wasn't able to deploy it.

It would be really useful if we could be able to restore per-db from a snapshot.

For a multi-tenant deployment this feature is extremely important.

All the best,

Rodrigo

Comment by yuanmei liu [ 25/Aug/16 ]

I am facing the same issue now. the backup strategy seems the same for WT with system snapshot. But can't restore the individual DB by copying individual folder. Even there is configuration folder per DB.

It's critical for us too.

Thanks,
Mei

Comment by Ramon Fernandez Marina [ 12/Apr/16 ]

I'm linking SERVER-19043, since I think the functionality described there could be used to implement this ticket.

Generated at Thu Feb 08 04:03:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.