[SERVER-32381] The copyDatabase shell helper should take a writeConcern option Created: 18/Dec/17  Updated: 30/Jul/19  Resolved: 30/Jul/19

Status: Closed
Project: Core Server
Component/s: Shell
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: asdf01 Assignee: Andrew Morrow (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to DOCS-11152 db.runCommand() accepts a writeConcer... Closed
related to SERVER-17392 db.copyDatabase helper should allow s... Closed
is related to SERVER-34885 Add ability to throttle or rate limit... Closed
Sprint: Dev Tools 2019-08-12
Participants:

 Description   

As the copyDatabase command can be resource intensive, there should be a way to pass writeConcern. The copydb command that copyDatabase uses internally does support writeConcern, so the shell helper should be able to pass that along.

Original Summary

Can you specify a write concern value on the copyDatabase operation

Original Description

When we copy large databases, we notice that a secondary in our 3 node replicate set goes into the RECOVERING state and never recovers.

I figured it's because the database we are copying is 8gb in size and the oplog isn't big enough. The replica set is configured to have a write concern of 1. So as soon as one node acknowledges the write, the Primary moves on. However these copy operations are progressing at such a pace that one of the secondaries isn't able to keep up and the master no longer has any oplog entries to allow the secondary to catch up.

Why this secondary doesn't fall back to the initial sync state I don't know. It just seems to stay in the RECOVERING state for weeks on end with no change to the optime at all.

I didn't want to change the default writeConcern to 2. Because then if one node goes down, then all the writes would fail waiting for a 2nd secondary to acknowledge the replication.

I didn't want to change the default writeConcern to "majority" based on the same fears. If one node goes down, is the majority condition determined by the majority of secondaries that are configured? Or the majority of the secondaries that are alive?

I thought it would be good if the copyDatabase operation takes a writeConcern value like the other db insert and update operations. Does mongodb do something like that? I couldn't find anything in the documentation.

Thanks



 Comments   
Comment by Andrew Morrow (Inactive) [ 30/Jul/19 ]

michael.qiu@wdtl.com - Please be aware that the copydb functionality is being removed in MongoDB 4.2, after its deprecation in 4.0. Given that, we will not be making changes to the shell helpers, especially given that there is an adequate workaround via the above runCommand discussion.

For additional information, please see https://docs.mongodb.com/master/release-notes/4.2-compatibility/#remove-support-for-the-copydb-and-clone-commands

I'm going to close this ticket now, but please follow up if we can provide any additional assistance.

Comment by Mark Agarunov [ 21/Dec/17 ]

Hello michael.qiu@wdtl.com,

Thank you for the additional information. Note that the writeConcern is an option passed to db.runCommand, not to copydb and can be passed to other commands executed with db.runCommand. However, as you noted neither the documentation for copydb nor db.runCommand mentions the writeConcern option so I've opened DOCS-11152 to address this issue.

Thanks,
Mark

Comment by asdf01 [ 19/Dec/17 ]

Hi mark.agarunov

Thanks for looking into this query.

Please note that SERVER project is for reporting bugs or feature suggestions

It is a feature suggestion: add write concern to the db.copyDatabase operation.

In this particular case, it looks like the feature already exists in the alternative copydb command. However no one outside the mongodb organisation is going to know about this feature if mongodb doesn't add it to its documentation. If you even bothered to look at the copydb page you referenced, you'll notice that there is no mention of writeConcern on there.

So instead of telling me to go away to some other forum, maybe your organization should be grateful that I brought this issue to your attention and take this opportunity to raise a documentation bug.

Thanks

Comment by Mark Agarunov [ 19/Dec/17 ]

Hello michael.qiu@wdtl.com,

Thank you for the report. To use writeConcern on the copyDatabase operation, you can execute the command with db.runCommand on the admin database using the copydb command (which db.runCommand is a wrapper around):

replset:PRIMARY> use admin
switched to db admin
replset:PRIMARY> db.runCommand({copydb:1, fromdb:'testing',todb:'testing2',writeConcern:{w:2}})
{ "ok" : 1 }

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thanks,
Mark

Comment by asdf01 [ 18/Dec/17 ]

Another way to avoid this problem is if we could throttle the copyDatabase operation some how. Is this possible? Thanks.

Generated at Thu Feb 08 04:30:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.