[SERVER-1311] mongodump/mongorestore loses information such as capped params Created: 28/Jun/10  Updated: 12/Jul/16  Resolved: 12/Dec/11

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 1.4.3
Fix Version/s: 2.1.0

Type: Bug Priority: Major - P3
Reporter: Roger Binns Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 13
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-808 mongodump should backup indexes when ... Closed
Operating System: ALL
Participants:

 Description   

Doing a dump, then a drop then a restore does not give you back what you had before. In particular meta-information about a collection such as it being capped and the capped size is not restored. I believe that paddingFactor is also lost although that won't be a functional difference like a collection being capped or not.



 Comments   
Comment by auto [ 12/Dec/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Log a warning when restoring with different options than existing collection. SERVER-1311.
Branch: master
https://github.com/mongodb/mongo/commit/3dd48a56ab7b0dbdbe2ce8ee1d3b9ee84c303b2b

Comment by auto [ 07/Dec/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Add test for dumping and restoring indexes and collection metadata. SERVER-1311
Branch: master
https://github.com/mongodb/mongo/commit/9122c89a7d58a8639e8625e1e4ed8767c4b7dd6b

Comment by auto [ 07/Dec/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Make mongorestore restore collection options and indexes from the metadata file. SERVER-1311
Branch: master
https://github.com/mongodb/mongo/commit/b54e8babe232ef2e9e5c71674b96c22855cc063b

Comment by auto [ 07/Dec/11 ]

Author:

{u'login': u'stbrody', u'name': u'Spencer T Brody', u'email': u'spencer@10gen.com'}

Message: Make mongodump create a metadata file with collection options and indexes to be used by mongorestore. SERVER-1311
Branch: master
https://github.com/mongodb/mongo/commit/a3aaded21e2f306714ebafd3a1e55ef175ce77fc

Comment by Spencer Brody (Inactive) [ 03/Aug/11 ]

If not using --drop check the system.namespaces collection and output a warning if the options don't match the options that were dumped.

Comment by Mathias Stearn [ 03/Aug/11 ]

Decided solution:

Create a collection_name.metadata.json file that has the following format:
{options: {}, indexes:[{}]}

options is optional and will only be there if the options field exists in db.system.namespaces.findOne(

{name:'db.collection'}

)
If it is, on restore make sure to update the name of the collection and run that object as a command.

indexes is a list of the objects in db.system.indexes.find(

{ns: 'db.collection'}

). On restore, after inserting the data for the collection, update the ns field and reinsert all the index objects.

Make sure mongorestore works even without a metadata.json file

Comment by Eliot Horowitz (Inactive) [ 01/Jun/11 ]

This might be a dupe

Comment by Roger Binns [ 28/Jun/10 ]

Actually it looks like I can do your solution instead, since I really don't want to hard code what attributes are normal/default for a collection. I was expecting system.namespaces to have a lot more crud in it. New algorithm:

  • For each doc in system.namespaces, if there are any keys other than name or options then output doc. If options contains any non-create key with non-null value then output that else exclude options For the vast majority of cases nothing would be output.
  • Output goes into system.namespaces.json (edit: same format as mongoexport - one json doc per line)
  • On restore use appropriate entry "options" key to create collection if it exists
Comment by Roger Binns [ 28/Jun/10 ]

Thanks. What I am going to implement is this:

  • If the directory contains a file named system.namespaces.* then leave everything up to mongorestore
  • If a collection.parm file exists then use it to create the collection

That should tide me over until Mongo has the official fix in, working with current and future versions.

BTW I would assume that having system.namespaces.bson would be really bad since older versions of mongorestore would try to restore it unilaterally. Using .json would avoid that issue.

Comment by Eliot Horowitz (Inactive) [ 28/Jun/10 ]

I guess we could make it system.namespaces.json instead of system.namespaces.bson
Don't want to add another extension (.parm) though

Comment by Eliot Horowitz (Inactive) [ 28/Jun/10 ]

I think we'll probably do something different.

Most likely we'll just write a system.namespaces.bson since that's what the name already is.

Then we'll just be smart about applying it.

Comment by Roger Binns [ 28/Jun/10 ]

Is my proposed solution acceptable? I need to implement something today, ideally the same or compatible with the eventual official fix.

Comment by Roger Binns [ 28/Jun/10 ]

I'd like to propose the following solution:

  • When doing a dump also create a collectionname.parm file alongside the .bson file. The contents are a single JSON object with parameters for creating the collection. This file only contains non-default parameters, and the whole file is omitted if it would otherwise be empty
  • When doing a restore, if the collection does not already exist (or was dropped) and there is a collectionname.parm file then the collection is created using those parameters

This will be forwards and backwards compatible with existing dumps and in the vast majority of cases the dumps will be identical.

Reasoning:

  • .parm extension is same length as .bson so that we don't hit any filename length issues
  • JSON is human readable
Generated at Thu Feb 08 02:56:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.