[SERVER-2771] Background index builds on replica set secondaries Created: 16/Mar/11 Updated: 18/Mar/16 Resolved: 01/May/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 2.5.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Richard Kreuter (Inactive) | Assignee: | Eric Milkie |
| Resolution: | Done | Votes: | 27 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
At present, index builds on replica set secondaries are always performed in the foreground, and so are blocking. So under normal circumstances, all ordinary secondaries (i.e., ones without slavedelay) will start building an index at about the same time, and so all of them will block together. |
| Comments |
| Comment by Eric Milkie [ 28/Mar/13 ] |
|
The fixVersion field, version 2.5.0, is the current target for this. We have one more thing to iron out before releasing this feature. |
| Comment by Colin Howe [ 28/Mar/13 ] |
|
Could we get an update on what is happening here? Some of the comments indicate that this was fixed in 2.2, some say 2.4 and then the status says 2.5. We just got bitten by this (again) after thinking that it was fixed 2.2.3... Pulling things out of replica sets to add indices really isn't a viable option. At least not for something that claims to make scaling easy with things like sharding support. |
| Comment by auto [ 25/Jan/13 ] |
|
Author: {u'date': u'2013-01-25T16:52:45Z', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: |
| Comment by Colin Howe [ 10/Jan/13 ] |
|
We just got hit by this... all our slaves stopped responding to queries and our clients just hung (no timeouts). Looking forward to seeing a fix! |
| Comment by auto [ 02/Jan/13 ] |
|
Author: {u'date': u'2013-01-02T15:41:44Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: This reverts commit e733331c7edb8871ba2dc58f20fbd2f40bf3adda. |
| Comment by auto [ 28/Dec/12 ] |
|
Author: {u'date': u'2012-12-28T20:05:56Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 27/Dec/12 ] |
|
Author: {u'date': u'2012-12-27T02:23:22Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 27/Dec/12 ] |
|
Author: {u'date': u'2012-12-27T01:42:47Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 27/Dec/12 ] |
|
Author: {u'date': u'2012-12-27T00:59:22Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 27/Dec/12 ] |
|
Author: {u'date': u'2012-12-27T00:53:37Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 26/Dec/12 ] |
|
Author: {u'date': u'2012-12-26T21:56:49Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 17/Dec/12 ] |
|
Author: {u'date': u'2012-12-17T16:39:26Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 13/Dec/12 ] |
|
Author: {u'date': u'2012-12-13T19:36:27Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 12/Nov/12 ] |
|
Author: {u'date': u'2012-11-12T16:58:14Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 12/Nov/12 ] |
|
Author: {u'date': u'2012-11-12T15:19:43Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by auto [ 12/Nov/12 ] |
|
Author: {u'date': u'2012-11-12T16:50:12Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: Revert " This reverts commit ab9ff1d3ecc7163fc93188efd997bbacd8fe7406. |
| Comment by auto [ 12/Nov/12 ] |
|
Author: {u'date': u'2012-11-12T15:19:43Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}Message: |
| Comment by Kristina Chodorow (Inactive) [ 18/Oct/12 ] |
|
Probably not, as there is a workaround (http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets, as you mentioned) and we'd like to concentrate on getting this totally done for 2.4. |
| Comment by Adam Flynn [ 17/Oct/12 ] |
|
I'd second Nathan's suggestion of adding an option to put the secondary into recovery state in this case as short-term solution to this problem since the fix didn't make it into 2.2. For any index build longer than a few seconds, I think killing the pending queries by going into recovery mode is a good trade-off for the extra operational overhead of doing what the docs propose currently or doing a replica set reconfig to go to hidden and back. Any chance of getting a change like that into 2.2.x while we wait for the secondary background indexing in 2.4? |
| Comment by auto [ 10/Oct/12 ] |
|
Author: {u'date': u'2012-10-10T13:18:34-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: |
| Comment by auto [ 10/Oct/12 ] |
|
Author: {u'date': u'2012-10-09T10:24:41-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}Message: |
| Comment by Eric Milkie [ 05/Sep/12 ] |
|
Implementation will proceed as per new technical spec. This will affect all background index builds (not just on secondaries). |
| Comment by Andy Schwerin [ 08/Jun/12 ] |
|
Correct handling of background operations on secondaries is going to require a more thorough examination of the concurrency model. |
| Comment by Scott Hernandez (Inactive) [ 07/Feb/12 ] |
|
On a unique, background, dropDups build we should do this synchronously since the order of the drops must be deterministic so replication isn't broken. |
| Comment by Kristina Chodorow (Inactive) [ 10/Nov/11 ] |
|
Index builds will now happen in the background on secondaries if they happen in the background on the primary. This means that reads can continue while the index builds. However, replication will block until the index build is complete. Please see http://www.mongodb.org/display/DOCS/Building+indexes+with+replica+sets#Buildingindexeswithreplicasets-Version2.1.0andlater for details. |
| Comment by auto [ 09/Nov/11 ] |
|
Author: {u'login': u'kchodorow', u'name': u'Kristina', u'email': u'kristina@10gen.com'}Message: Allow background indexes while in secondary state |
| Comment by Mathias Stearn [ 31/Oct/11 ] |
|
Actually, I think we do want it blocking replication at least at first so that the getLastError( {w:'all'}) semantics can be relied on to wait until the index is built on all nodes. This will allow the slave to fall behind, but this is a current issue with building in forground so at least it isn't a regression (other than being slower so increased likely hood of falling out of sync). Since we know there will be no other writes to the slave while the index is building would it be possible to do an optimized background build, or maybe even a foreground build that yields or fakes it by using a read lock? |
| Comment by Dwight Merriman [ 31/Oct/11 ] |
|
another case to analyze is that well into the bg create on the primary, a killOp stops the operation. as it were likely already placed in oplog, that would be an issue. |
| Comment by Nathan Acuff [ 30/Oct/11 ] |
|
An index build that takes longer than a couple of seconds is worse. Sounds like it's not going to be an issue though, and that's great. |
| Comment by Eliot Horowitz (Inactive) [ 29/Oct/11 ] |
|
Nathan - that could be bad if people are querying. |
| Comment by Nathan Acuff [ 28/Oct/11 ] |
|
Would it be possible to have the secondary go into a RECOVERING state, like it does when compacting? This would let you build indexes in a pinch without sprinkling rs.reconfig everywhere. The current behavior is much worse. |
| Comment by Jeff Yemin (Inactive) [ 01/Aug/11 ] |
|
Can someone explain why the master can do background indexing but secondaries can't? Why the need to make a secondary hidden just to build an index? |
| Comment by Dwight Merriman [ 29/Jul/11 ] |
|
one workaround would be to reconfigure with the secondary member having hidden : true, then create the index, background on primary and foreground on secondary but secondary is then effectively offline for clients. then afterwards go back to original replica set configuration. one detail here is that after reconf to hidden the secondary mongod probably needs to be restarted to make it drop existing client connections. |
| Comment by Dwight Merriman [ 29/Jul/11 ] |
|
nuances:
|
| Comment by Robert Fan [ 17/Mar/11 ] |
|
Gotcha, thanks Eliot. Doubly for your late-nightedness JIRA mastery |
| Comment by Eliot Horowitz (Inactive) [ 17/Mar/11 ] |
|
@robert - this is basically a new feature, so highly unlikely to be backported to 1.8. |
| Comment by Robert Fan [ 17/Mar/11 ] |
|
Would love to see this on 1.8.x if at all possible. We create several new collections on a daily basis and add about 5 indices per collection created. With our current size (~1TB) adding an index after-the-fact takes about 15 hours and we can't afford to have the secondaries out of commission for so long. |
| Comment by Richard Kreuter (Inactive) [ 16/Mar/11 ] |
|
Is the issue that because only one background index build can happen at a time, if the secondary started a background index build and encounters a subsequent background build op in the oplog, it'd have to either pause replication or do something that amounts to enqueuing the second index build for later processing? If that's the only issue, then having a persisted queue of to-be-constructed indexes and a background thread that processes the queue could also mean that createIndex wouldn't have to fail while an index was under construction even in non-RS deployments. |
| Comment by Dwight Merriman [ 16/Mar/11 ] |
|
if( background && cc().isSyncThread() ) { /* don't do background indexing on slaves. there are nuances. this could be added later but requires more code. */ log() << "info: indexing in foreground on this replica; was a background index build on the primary" << endl; background = false; } |
| Comment by Dwight Merriman [ 16/Mar/11 ] |
|
there was a reason why it was incorrect to do the background build on the secondaries. we should fix it. |