[SERVER-25297] mongorestore: oplog + capped collection problem Created: 27/Jul/16  Updated: 31/May/17  Resolved: 02/May/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Anton Assignee: Kelsey Schubert
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

When I try restore databases from dump, mongorestore gives an error:

2016-07-27T14:51:07.204+0700    restoring indexes for collection dbname.ip_stats from metadata
2016-07-27T14:52:12.049+0700    finished restoring dbname.ip_stats (10805669 documents)
2016-07-27T14:52:12.051+0700    restoring users from dbname/dump/admin/system.users.bson.gz
2016-07-27T14:52:12.076+0700    restoring roles from dbname/dump/admin/system.roles.bson.gz
2016-07-27T14:52:12.151+0700    error dropping temporary collection admin.temproles: ns not found
2016-07-27T14:52:12.151+0700    replaying oplog
2016-07-27T14:52:13.604+0700    Failed: restore error: error applying oplog: applyOps: Cannot change the size of a document in a capped collection: 222 != 218

dump created by: mongodump with --oplog --gzip options
same options passed in mongorestore



 Comments   
Comment by Kelsey Schubert [ 02/May/17 ]

Hi Hett,

We have manually inspected the files you have uploaded and created a custom build to help us debug this issue. Unfortunately, we are unable to reproduce this behavior. If you're able to provide a script that readily reproduces this issue across environments, please let us know and we will continue to investigate.

Thank you,
Thomas

Comment by Anton [ 18/Oct/16 ]

that's how I did dump:

mongodump --out /backup/nfs/1/ --oplog --host rs_name/data_node1,data_node2,data_node3 --username sa --password secret --authenticationDatabase admin --gzip

Comment by Anton [ 05/Oct/16 ]

Hi,
1. Dump created from Primary node.
2. Now I upgrade all replica members (including tools) to version 3.2.10, create dump and try to restore:

2016-10-05T09:03:54.723+0000    restoring indexes for collection k2k.node_threads_reduced from metadata
2016-10-05T09:10:24.665+0000    finished restoring k2k.node_threads_reduced (14848171 documents)
2016-10-05T09:10:24.675+0000    restoring users from /mnt/backup/1/admin/system.users.bson.gz
2016-10-05T09:10:24.924+0000    restoring roles from /mnt/backup/1/admin/system.roles.bson.gz
2016-10-05T09:10:26.441+0000    error dropping temporary collection admin.temproles: ns not found
2016-10-05T09:10:26.442+0000    replaying oplog
2016-10-05T09:10:29.458+0000    oplog   18.7KB
2016-10-05T09:10:32.458+0000    oplog   38.5KB
2016-10-05T09:10:33.763+0000    oplog   46.7KB
2016-10-05T09:10:33.768+0000    Failed: restore error: error applying oplog: applyOps: Cannot change the size of a document in a capped collection: 165 != 161

Comment by Kelsey Schubert [ 03/Oct/16 ]

Hi Hett,

Thanks for the additional information. Unfortunately, the use of the Timestamp data type would not explain this issue.

To help our investigation, I have few follow up questions:

  1. Was the node you were dumping from a Primary or Secondary?
  2. Have you observed this issue on subsequent mongodumps and mongorestores?

Thank you,
Thomas

Comment by Anton [ 20/Sep/16 ]

Hello!

Just I found in our application usage Timestamp datatype instead of ISODate.
Documentation say that is internal type:

>The BSON timestamp type is for internal MongoDB use. For most cases, in application development, you will want to use the BSON date type. See Date for more information.

Maybe this was the reason for the failure?

Comment by Anton [ 12/Aug/16 ]

Hi,

>Please answer the following questions so we can continue to investigate:
1. our production replica set contain only nodes with version 3.2.8 and using engine WT. I check it again. Dump created by mongodump has the same version. I check it repeatedly too.
2. as far as I remember - yes
3. Unfortunately i can not provide full version of database by our corp reasons. Howerer i found problematic collection and we can provide it with other collections metadata. This is enough to reproduce the problem. Data was uploaded on secure portal (oplog_plus_one_collection_and_all_metadata.zip)

mongorestore --oplogReplay --gzip --drop --dir=dump/dump --verbose --host=10.0.0.3
....
2016-08-12T18:11:03.596+0700    replaying oplog
2016-08-12T18:11:04.681+0700    Failed: restore error: error applying oplog: applyOps: Cannot change the size of a document in a capped collection: 103 != 118

Sorry for my english.

Comment by Kelsey Schubert [ 09/Aug/16 ]

Hi Hett,

Thank you for uploading the oplog of the dump, however, to continue to debug this issue we will the complete dump, which will contain the objects in their initial size and create the collections with the appropriate settings.

My understanding is that the data was dumped from a 3.2.8 node and was attempted to be restored into a 3.2.8 node as well. If that is not the case, what you are observing may be expected.

In MongoDB 3.2, if an update changes the document size, the operation will fail. In MongoDB 3.0, with MMAPv1 only if the update operation causes a document to grow beyond the document’s original size would the update operation would fail, and with WiredTiger there was no restriction on objects in capped collections (SERVER-20529).

Please answer the following questions so we can continue to investigate:

  • Would you please confirm that the dump was taken from a 3.2.x mongod?
  • Are you able to successfully restore into an older version of MongoDB?
  • Would you please upload the complete dump to the same upload portal as before?

Thanks again,
Thomas

Comment by Anton [ 27/Jul/16 ]

Ramon Fernandez,
anywhere versions is 3.2.8

mongod --version
db version v3.2.8
git version: ed70e33130c977bda0024c125b56d159573dbaf0
 
# mongodump --version
mongodump version: r3.2.8
git version: 2214d4d6561574f962c1dc72fefce4fe11843023
 
root@ubuntu:~# mongorestore --version
mongorestore version: r3.2.8
git version: 2214d4d6561574f962c1dc72fefce4fe11843023

I uploaded oplog on secure portal.

Comment by Ramon Fernandez Marina [ 27/Jul/16 ]

Hett, what versions of mongod and mongodump did you use to dump the data? SERVER-20529 describes a bug that may be the cause of the behavior you described, so I'd like to first determine if it's the same issue you're seeing or if this is a completely new problem.

If this is all 3.2.8, it could be helpful for use to have access to the data dump itself; I've created a secure upload portal where you can upload the file(s) created by mongodump – please let us know if you can upload your oplog so we can investigate further.

Thanks,
Ramón.

Comment by Anton [ 27/Jul/16 ]

pc1# mongodump --oplog --host 127.0.0.1 --username sa --password secret --authenticationDatabase admin --gzip
pc2# mongorestore --drop --oplogReplay --gzip --verbose --dir=data/dump

Generated at Thu Feb 08 04:08:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.