[SERVER-3937] Allow mongorestore to be throttled Created: 23/Sep/11  Updated: 11/Jul/16  Resolved: 13/Feb/12

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: None
Fix Version/s: 2.1.0

Type: Bug Priority: Major - P3
Reporter: Kristina Chodorow (Inactive) Assignee: Brandon Diamond
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Participants:

 Description   

It can restore docs faster than they can be replicated, leading to stale slaves. Maybe something like a --getLastError option so you could pass in

{w:3}

or something and it would do that periodically.



 Comments   
Comment by auto [ 14/Feb/12 ]

Author:

{u'login': u'', u'name': u'Brandon Diamond', u'email': u'brandon@10gen.com'}

Message: SERVER-3937: recommit of restore throttling test

Commit by another contributor accidentally clobbered earlier changes.
Branch: master
https://github.com/mongodb/mongo/commit/35cdfe783700a5ce95805d18b30402374ec4cdf4

Comment by Brandon Diamond [ 13/Feb/12 ]

Added basic test ensuring that mongorestore works with write concern (though the test does not verify that the write concern itself is respected).

By inspection, running with write concern set is much slower than running without – which seems to indicate that the process is throttled as expected.

Comment by auto [ 13/Feb/12 ]

Author:

{u'login': u'', u'name': u'Brandon Diamond', u'email': u'brandon@10gen.com'}

Message: SERVER-3937: Basic test for mongorestore with wc
Branch: master
https://github.com/mongodb/mongo/commit/11578fe802e6c2b990cb47e7782182cfbd5096ca

Comment by auto [ 11/Oct/11 ]

Author:

{u'login': u'', u'name': u'Brandon Diamond', u'email': u'brandon@10gen.com'}

Message: SERVER-3937: Only call GLE if w > 1
Branch: master
https://github.com/mongodb/mongo/commit/31bb33c1207a288c49748c869681d96be808622b

Comment by Eliot Horowitz (Inactive) [ 10/Oct/11 ]

Yes - only call getLastError at all if w > 1

Comment by Brandon Diamond [ 10/Oct/11 ]

This can be added quite easily; rather than issuing the calls on w <= 1, we can simply skip that logic. There's still a slowdown for the branch, but that's nothing compared to the GLE call. Shall I?

Comment by Spencer Brody (Inactive) [ 10/Oct/11 ]

This doesn't look like this provides a way to do the import without calling getLastError at all, in the cases when fire-and-forget writes are fine. This could wind up slowing down imports that don't need the write concern. There should also probably be an option to continue the import when some of the inserts fail (similar to the continueOnError flag for bulk inserts).

Comment by auto [ 07/Oct/11 ]

Author:

{u'login': u'', u'name': u'Brandon Diamond', u'email': u'brandon@10gen.com'}

Message: SERVER-3937: Added write concern support to mongoimport
Branch: master
https://github.com/mongodb/mongo/commit/6fee908ce602f3f6b7763f21ca35dca3766d7443

Comment by Brandon Diamond [ 06/Oct/11 ]

I'm going to take a crack at this today. Any decision on the options to be supported?

Comment by Scott Hernandez (Inactive) [ 26/Sep/11 ]

It does seem natural that you could pass in "

{w:2}

" or "

{w:'majority', wtimeout:500}

".

Comment by Kristina Chodorow (Inactive) [ 26/Sep/11 ]

What about j? (And we're only going to have more getLastError options in the future.) I think that, if we want to minimize number of options, we should let people pass in any getLastError args (not just w).

Comment by Eliot Horowitz (Inactive) [ 23/Sep/11 ]

More options != good.
We should start with just --w and no frequency option, just choose something

Comment by Scott Hernandez (Inactive) [ 23/Sep/11 ]

It would be good to able to specify all gle options, and frequency in doc or bytes

Comment by Eliot Horowitz (Inactive) [ 23/Sep/11 ]

Maybe just --w 2
So you can specify actually how many you want it go to

Generated at Thu Feb 08 03:04:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.