[SERVER-10630] Speed of cleanupOldData while chunk balancing Created: 27/Aug/13 Updated: 11/Jul/16 Resolved: 07/Nov/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.4.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Steffen | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 12.04.1 LTS |
||
| Operating System: | ALL |
| Participants: |
| Description |
|
We started to shard one more of our big collections in our database. Database has 26 collections and some of them are already sharded.
The collection we now added has around 140 mio documents. What we now see, is that outside the Balancer window the homeshard is doing its cleanup rounds. Thus we see a lot writes and reads via mongotop on this collection. some output dbtop (webinterface) for this collection:
In the logfile of the server process (primary) we find the following entry:
Every cleanup deletes around 90k documents in ~24 minutes. This is very slow and we suffer from periodic high IO writes. During these high IO writes the mongod service is slow and we queue up reads and some writes (monitored via mongostat) Is this cleanUp job so aggressive for the IO? Thanks in advance, |
| Comments |
| Comment by ning [ 14/Nov/13 ] | ||||||||||||||||||||||||||||||||||
|
can we set speed of remove records in cleanupOldData. | ||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 07/Nov/13 ] | ||||||||||||||||||||||||||||||||||
|
Hi Steffen, it looks like the issues outlined in this ticket have been addressed, so I'm resolving as fixed. Please feel to re-open if you experience related issues again. | ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 19/Sep/13 ] | ||||||||||||||||||||||||||||||||||
|
We found an Issue with our Repset setup and a Arbiter. This leads to the following problems.
This Function calculates in our setup to a result of 3. This means that every physical Machine in our repset needs to be up2date in replication lag.
We hit this for every moveChunk operation. | ||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 02/Sep/13 ] | ||||||||||||||||||||||||||||||||||
|
Hi Steffen, Regarding the 4 member replica set, it doesn't hurt so much as it doesn't really help. If you lose 2 members from a 4 member or a 3 member replica set, the result is the same, your set will not be able to receive writes. See: http://docs.mongodb.org/manual/core/replica-set-architecture-three-members/ | ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 30/Aug/13 ] | ||||||||||||||||||||||||||||||||||
|
FYI we are using tag aware sharding to force where the sharded collection remain.
So all our sharded databases/collections are on shards 5-10 only. | ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 30/Aug/13 ] | ||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 30/Aug/13 ] | ||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 29/Aug/13 ] | ||||||||||||||||||||||||||||||||||
|
Is there a way to specify the write concern for _secondaryThrottle ?
| ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 29/Aug/13 ] | ||||||||||||||||||||||||||||||||||
|
BTW, there is syntax error in the documentation at http://docs.mongodb.org/manual/tutorial/configure-sharded-cluster-balancer/#require-replication-before-chunk-migration-secondary-throttle Wrong bracket on $set close:
Working:
| ||||||||||||||||||||||||||||||||||
| Comment by Steffen [ 29/Aug/13 ] | ||||||||||||||||||||||||||||||||||
|
It's the shard "repset2". This is the homeshard of the database with name refind. | ||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 29/Aug/13 ] | ||||||||||||||||||||||||||||||||||
|
Hi Stefan, There are a couple settings which control the aggressiveness of chunk migrations and cleanup. Starting in v2.4 the _secondaryThrottle option was turned on by default to send each insert and delete with a write concern of w:2 to try and mitigate shard migration and cleanup costs. See: http://docs.mongodb.org/manual/tutorial/configure-sharded-cluster-balancer/#require-replication-before-chunk-migration-secondary-throttle Also, starting 2.2.1, migration cleanups are performed asynchronously to allow migrations to continue, while the old data is still being deleted from the donor shard. This can result in deletions "leaking" past the balancer window time. I see you have MMS monitoring enabled. Can you identify a particular shard where this issue is occurring? |