[SERVER-19043] Implement a connectDatabase / disconnectDatabase functionality. Created: 18/Jun/15 Updated: 21/Aug/20 Resolved: 21/Aug/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Paul Reed | Assignee: | Brian Lane |
| Resolution: | Done | Votes: | 14 |
| Labels: | asya | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Windows for me, but probably the other places. |
||
| Issue Links: |
|
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||
| Description |
|
With the old engine, you could shift folders/files in and out and they would become immediately available for access from clients. Is it possible to generate this missing structure simply by issueing a command which would then parse the file structure to rebuild its internal mappings. Hope that explains the requirement. Sorry if this request is already somewhere else - its a little fiddly to find through Jira. |
| Comments |
| Comment by Brian Lane [ 21/Aug/20 ] |
|
As we discussed, I will go ahead and close this issue as gone away. As you continue to experiment with mtransfer, feel free to reach out to me directly if you have questions or encounter any problems. |
| Comment by Brian Lane [ 18/Feb/20 ] |
|
We have recently released an updated version of mtools, which includes a new tool called mtransfer that should help address some of the requirements raised in this issue. We have currently flagged it as experimental as we gather feedback. Feel free to reach out to me directly via brian.lane@mongodb.com if you encounter any issues getting mtools setup in your environment or have any feedback regarding mtransfer. |
| Comment by Brian Lane [ 08/Oct/19 ] |
|
Unfortunately not. I will reach out to you via email to set up a meeting to discuss. I want to get you to a later version w/ WiredTiger, but don't want this issue to be the blocker. -Brian |
| Comment by Paul Reed [ 02/Oct/19 ] |
|
Status is still Open on this one. Is there any movement forward with a solution which would work, and thus allow me to move to the latest version. Any update ? |
| Comment by Matt Hughes [ 11/May/19 ] |
|
On WT here with version 3.6.12 and would very much like to see this functionality, in 3.6 if possible as we are not ready to move to 4.x. I have terabytes of data and cannot dump to the destination and import due to storage restrictions, was very surprised to see this is not possible. This would be invaluable in recovery situations. |
| Comment by Paul Reed [ 23/Apr/19 ] |
|
Currently we are sitting at : MongoDB server version: 3.6.2 on the MMAPv1. This is the only issue at the moment preventing us moving to WT. The worry of corruption requiring a full architecture rebuild as noticed in some posts ( maybe in the past and not current ) is also a worry, however, the mechanics with directory backups somewhat negate that risk as well as allow us greater data update management.
|
| Comment by Brian Lane [ 23/Apr/19 ] |
|
Hi paul.reed, I can't make a guarantee at this time regarding implementation. However, we have made some changes in the 4.2 release with backup cursors that could make the work required here potentially not as bad. What version of the server are you currently using? I assume you are still with MMAPv1 as well? Is this the main issue preventing you from migrating to a newer version w/ WiredTiger? -Brian |
| Comment by Paul Reed [ 15/Apr/19 ] |
|
Or at the least allow for the metadata of a database to reside in the db folder (therefore allowing folder copying) |
| Comment by Paul Reed [ 15/Apr/19 ] |
|
Just seen you've changed status. Please please please implement this. |
| Comment by Paul Reed [ 04/Jul/18 ] |
|
Please can you revisit this item now that MMAPv is deprecated. |
| Comment by Chris Kuethe [ 15/Jun/17 ] |
|
This would also be helpful for emergency recovery situations: copy in the database directory to the root of a new dbpath, start up a new server, and attach and repair it. |
| Comment by Konstantin Manchev [ 26/Nov/16 ] |
|
I do not understand how such critical issue has only 9 votes up to now with mine from today ... , please, consider this seems to be a serious limitation in flexibility in MongoDB/WiredTiger ... |
| Comment by Yuri Finkelstein [ 05/May/16 ] |
|
I can't agree more with Paul. No matter how great are WT concurrency improvements are, the fact is that when one was able to build an image of a database offline on another machine, upload the entire directory to every member of the replica set, and just "activate" it by telling application to start writes to this database it was 100 times more efficient and predictable compared to mongorestore. For one, the secondaries don't need to replicate the new DB which is the case with mongorestore! The user can throttle the process of file copying to RS members and achieve smooth and predictable outcome. I just don't understand why mongodb is not seeing the merits of this process and instead is trying to convince customers than WT is better than mmapv1. This issue is not at all about what engine is better. |
| Comment by Daniel Pasette (Inactive) [ 06/Jan/16 ] |
|
Hi Paul, A couple questions:
Also, there is the possibility that you could continue doing your processing on the side and simply use mongorestore to import the data into your prod servers. There are a few knobs you can use to control the level of concurrency, and thus load, with this approach. |
| Comment by Paul Reed [ 05/Jan/16 ] |
|
Thanks for replying. I am inferring the issue from experiences with users being locked out of access through MMAPv1. I need to rebuild sandbox areas from datasnapshots 2 or 3 times a month from subsets of imported data. I have found that aggregation on large datasets will cause locking at db level, which causes our users to block on reads. Also, the large rebuild, which by necessity drops and recreates grouped up data, including lengthy index builds, would cause areas of data to go missing from sight. I suppose building into a temporary database and then swapping users to it in a versioned sort of way might be the work around, which I have not tried. The sheer convenience of backup and restore processes based on directory level copy was a massive factor in our using Mongo as our data store. Paul |
| Comment by Daniel Pasette (Inactive) [ 05/Jan/16 ] |
|
Hi Paul, I understand your use case but this is not a small change to WiredTiger and the MongoDB integration layer and we have no immediate plans to add this feature to MongoDB with WiredTiger. Have you tested the impact of the data import on your system with WiredTiger, or are you inferring the issue from experience with MMAPv1? Thanks, |
| Comment by Paul Reed [ 05/Jan/16 ] |
|
Any further thoughts on this: really would love for the meta to be stored in the database directory when directory level databases are enabled. |
| Comment by Paul Reed [ 03/Dec/15 ] |
|
I wonder if there is any further thoughts on adding this re-sync meta process to allow for folder migration in and out of the WiredTiger engine. I am desperate to get to the benefits of being able to copy collection level files around replicasets (at the moment I copy the entire database with the MM engine, which is ok - but could be optimised better at collection level with WT) and also to get the speed benefits from WT. |
| Comment by Ramon Fernandez Marina [ 05/Aug/15 ] |
|
There is currently no process to manually move files around or recreate the necessary metadata when using the WiredTiger engine. This ticket remains open to consider this functionality in future releases. Cheers, |
| Comment by Paul Reed [ 04/Aug/15 ] |
|
So is there a process currently, or could there be a process to force the rebuild of this meta. Or possibly could the meta be stored in the directory - for directory based databases - and therefore folder copy would be doable again ? |
| Comment by Michael Cahill (Inactive) [ 31/Jul/15 ] |
|
This procedure won't work with WiredTiger regardless of sizeStorer: "copying over the database folder" will just lead to corruption because the WiredTiger metadata won't match the files. |
| Comment by Ramon Fernandez Marina [ 30/Jul/15 ] |
|
paul.reed, my guess is that the moment your primary takes a single write operation during this process the sizeStorer.wt file may become out of sync (I'm also guessing this may happen after a checkpoint too), which would negate the benefits of the procedure. I'll let others with more WiredTiger knowledge comment, but in the meantime if you're to try this procedure I'd strongly recommend you do it in a test setup. |
| Comment by Paul Reed [ 30/Jul/15 ] |
|
If I did this, would it work around the metadata missing problem: 1) Against RS: create new database ( create all new possible collections with a {} insert ) |
| Comment by Paul Reed [ 30/Jul/15 ] |
|
What information is stored in the metadata ? |
| Comment by Ramon Fernandez Marina [ 19/Jun/15 ] |
|
Thanks for the additional information paul.reed. Apologies if I wasn't clear: we do not consider this a WiredTiger issue because is not about using WiredTiger stand-alone, but as a storage engine inside MongoDB. If you are using WiredTiger without MongoDB then we should move this ticket back to the WT project, but as long as is related to MongoDB it should remain in SERVER with the rest of the MongoDB issues. Hope that clarifies things. WiredTiger keeps additional metadata in files inside dbpath, so as you've found collection files are not detected when dropped in. I'm tagging this ticket as "Needs Triage" for further consideration. Thanks, |
| Comment by Paul Reed [ 19/Jun/15 ] |
|
Well: |
| Comment by Ramon Fernandez Marina [ 18/Jun/15 ] |
|
paul.reed, I've moved this ticket to the SERVER project, as my understanding of your description is that this is really a MongoDB issue. Can you please elaborate on the "shift folders/files" part of your description above and provide a recipe/scenario where this was handy in the past? That should help us understand your suggestion better. Thanks, PS: yes, some times is hard to find things in JIRA |
| Comment by Paul Reed [ 18/Jun/15 ] |
|
Also - folder copying very easily created a fallback safety net - for sensitive operations. This would be even better with the WT way of naming files based around the collection - rather than the database. |