[SERVER-2771] Background index builds on replica set secondaries Created: 16/Mar/11  Updated: 18/Mar/16  Resolved: 01/May/13

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.5.0

Type: Improvement Priority: Major - P3
Reporter: Richard Kreuter Assignee: Eric Milkie
Resolution: Fixed Votes: 27
Labels: None

Issue Links:
Depends
depends on SERVER-9452 index rebuilder can't use DBDirectCli... Closed
is depended on by PYTHON-202 Hang when calling conn.database_names... Closed
is depended on by SERVER-4820 renaming a collection during bg index... Closed
Duplicate
is duplicated by SERVER-1645 could be issues with replication of b... Closed
Gantt Dependency
has to be done after SERVER-8536 reenable IndexRebuilder for backgroun... Closed
Related
related to SERVER-2374 Should be able to run multiple backgr... Closed
related to SERVER-4662 Building large index causes primary t... Closed
related to SERVER-9421 queryoptimizer3.js failing during par... Closed
related to SERVER-6836 Allow index builds on secondaries wit... Open
related to SERVER-6883 index creation on secondaries need no... Open
is related to SERVER-14746 IndexRebuilder should only restart in... Open
is related to SERVER-4820 renaming a collection during bg index... Closed
is related to SERVER-8044 Possible deadlock in killOp/curOp Closed
is related to SERVER-10460 Index Rebuilder may not always run at... Closed
is related to SERVER-11219 mongod crash during index build leave... Closed
is related to SERVER-12727 index building can make replica set m... Closed
is related to DOCS-342 Background index behaviour on a secon... Closed
is related to DOCS-739 Document --noIndexBuildRetry option Closed
Participants:

 Description   

At present, index builds on replica set secondaries are always performed in the foreground, and so are blocking. So under normal circumstances, all ordinary secondaries (i.e., ones without slavedelay) will start building an index at about the same time, and so all of them will block together.



 Comments   
Comment by Dwight Merriman [ 16/Mar/11 ]

there was a reason why it was incorrect to do the background build on the secondaries.

we should fix it.

Comment by Dwight Merriman [ 16/Mar/11 ]

if( background && cc().isSyncThread() )

{ /* don't do background indexing on slaves. there are nuances. this could be added later but requires more code. */ log() << "info: indexing in foreground on this replica; was a background index build on the primary" << endl; background = false; }
Comment by Richard Kreuter [ 16/Mar/11 ]

Is the issue that because only one background index build can happen at a time, if the secondary started a background index build and encounters a subsequent background build op in the oplog, it'd have to either pause replication or do something that amounts to enqueuing the second index build for later processing?

If that's the only issue, then having a persisted queue of to-be-constructed indexes and a background thread that processes the queue could also mean that createIndex wouldn't have to fail while an index was under construction even in non-RS deployments.

Comment by Sharethrough Engineering [ 17/Mar/11 ]

Would love to see this on 1.8.x if at all possible. We create several new collections on a daily basis and add about 5 indices per collection created. With our current size (~1TB) adding an index after-the-fact takes about 15 hours and we can't afford to have the secondaries out of commission for so long.

Comment by Eliot Horowitz [ 17/Mar/11 ]

@robert - this is basically a new feature, so highly unlikely to be backported to 1.8.
2.0 should be out in late q2

Comment by Sharethrough Engineering [ 17/Mar/11 ]

Gotcha, thanks Eliot. Doubly for your late-nightedness JIRA mastery Take care, Rob

Comment by Dwight Merriman [ 29/Jul/11 ]

nuances:

  • first the caller to background index doesn't return until that finishes, even though other things do continue. so if done in the initial sync thread it will likely stop until built even though other readers can keep going.
  • if there are unique key constraints, will the behavior be nondeterministic relative to the primary? i think so? one incremental step would be to do background for non-unique indexes as a start
  • what else that i'm forgetting?
Comment by Dwight Merriman [ 29/Jul/11 ]

one workaround would be to reconfigure with the secondary member having hidden : true, then create the index, background on primary and foreground on secondary but secondary is then effectively offline for clients. then afterwards go back to original replica set configuration.

one detail here is that after reconf to hidden the secondary mongod probably needs to be restarted to make it drop existing client connections.

Comment by Jeff Yemin [ 01/Aug/11 ]

Can someone explain why the master can do background indexing but secondaries can't? Why the need to make a secondary hidden just to build an index?

Comment by Nathan Acuff [ 28/Oct/11 ]

Would it be possible to have the secondary go into a RECOVERING state, like it does when compacting? This would let you build indexes in a pinch without sprinkling rs.reconfig everywhere. The current behavior is much worse.

Comment by Eliot Horowitz [ 29/Oct/11 ]

Nathan - that could be bad if people are querying.
This will be fixed for 2.2 so the index can be done in the background like on the primary.

Comment by Nathan Acuff [ 30/Oct/11 ]

An index build that takes longer than a couple of seconds is worse.

Sounds like it's not going to be an issue though, and that's great.

Comment by Dwight Merriman [ 31/Oct/11 ]

another case to analyze is that well into the bg create on the primary, a killOp stops the operation. as it were likely already placed in oplog, that would be an issue.

Comment by Mathias Stearn [ 31/Oct/11 ]

Actually, I think we do want it blocking replication at least at first so that the getLastError(

{w:'all'}

) semantics can be relied on to wait until the index is built on all nodes. This will allow the slave to fall behind, but this is a current issue with building in forground so at least it isn't a regression (other than being slower so increased likely hood of falling out of sync).

Since we know there will be no other writes to the slave while the index is building would it be possible to do an optimized background build, or maybe even a foreground build that yields or fakes it by using a read lock?

Comment by auto [ 09/Nov/11 ]

Author:

{u'login': u'kchodorow', u'name': u'Kristina', u'email': u'kristina@10gen.com'}

Message: Allow background indexes while in secondary state SERVER-2771
Branch: master
https://github.com/mongodb/mongo/commit/72dcc0b296306fc432f52ada8eedf4814c791b68

Comment by Kristina Chodorow [ 10/Nov/11 ]

Index builds will now happen in the background on secondaries if they happen in the background on the primary. This means that reads can continue while the index builds. However, replication will block until the index build is complete.

Please see http://www.mongodb.org/display/DOCS/Building+indexes+with+replica+sets#Buildingindexeswithreplicasets-Version2.1.0andlater for details.

Comment by Scott Hernandez [ 07/Feb/12 ]

On a unique, background, dropDups build we should do this synchronously since the order of the drops must be deterministic so replication isn't broken.

Comment by Andy Schwerin [ 08/Jun/12 ]

Correct handling of background operations on secondaries is going to require a more thorough examination of the concurrency model.

Comment by Eric Milkie [ 05/Sep/12 ]

Implementation will proceed as per new technical spec. This will affect all background index builds (not just on secondaries).

Comment by auto [ 10/Oct/12 ]

Author:

{u'date': u'2012-10-09T10:24:41-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-2771 preliminary code cleanup; no functional change
Branch: master
https://github.com/mongodb/mongo/commit/7922e03d36d059a6602541d83b6a26224e6178dd

Comment by auto [ 10/Oct/12 ]

Author:

{u'date': u'2012-10-10T13:18:34-07:00', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-2771 add synchronous kill functionality for background indexing support
Branch: master
https://github.com/mongodb/mongo/commit/800eea522786a5a80cec62a6eafc382b3bc377bf

Comment by Adam Flynn [ 17/Oct/12 ]

I'd second Nathan's suggestion of adding an option to put the secondary into recovery state in this case as short-term solution to this problem since the fix didn't make it into 2.2. For any index build longer than a few seconds, I think killing the pending queries by going into recovery mode is a good trade-off for the extra operational overhead of doing what the docs propose currently or doing a replica set reconfig to go to hidden and back. Any chance of getting a change like that into 2.2.x while we wait for the secondary background indexing in 2.4?

Comment by Kristina Chodorow [ 18/Oct/12 ]

Probably not, as there is a workaround (http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets, as you mentioned) and we'd like to concentrate on getting this totally done for 2.4.

Comment by auto [ 12/Nov/12 ]

Author:

{u'date': u'2012-11-12T15:19:43Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Retry index builds on startup
Branch: master
https://github.com/mongodb/mongo/commit/ab9ff1d3ecc7163fc93188efd997bbacd8fe7406

Comment by auto [ 12/Nov/12 ]

Author:

{u'date': u'2012-11-12T16:50:12Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: Revert "SERVER-2771 Retry index builds on startup"

This reverts commit ab9ff1d3ecc7163fc93188efd997bbacd8fe7406.
Branch: master
https://github.com/mongodb/mongo/commit/3abcc53d43145bcda52016c753e10ff139d41e4d

Comment by auto [ 12/Nov/12 ]

Author:

{u'date': u'2012-11-12T15:19:43Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Retry index builds on startup
Branch: master
https://github.com/mongodb/mongo/commit/ab1bceba74845666aaad957cab256cadf2a0c8de

Comment by auto [ 12/Nov/12 ]

Author:

{u'date': u'2012-11-12T16:58:14Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Fix tests that use addIndex
Branch: master
https://github.com/mongodb/mongo/commit/d9df6423f805b5cd1dc7717b62a13aae9ce3778a

Comment by auto [ 13/Dec/12 ]

Author:

{u'date': u'2012-12-13T19:36:27Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Multiple index builds
Branch: master
https://github.com/mongodb/mongo/commit/2812b757dd25de5ca22c3c84135f3c654ffcb928

Comment by auto [ 17/Dec/12 ]

Author:

{u'date': u'2012-12-17T16:39:26Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Add debugging to test
Branch: master
https://github.com/mongodb/mongo/commit/41cce287ffad7ea0c04facdd0986631bade7d027

Comment by auto [ 26/Dec/12 ]

Author:

{u'date': u'2012-12-26T21:56:49Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Command handling
Branch: master
https://github.com/mongodb/mongo/commit/2f21426abe2d75487c91fb361154261969d303cd

Comment by auto [ 27/Dec/12 ]

Author:

{u'date': u'2012-12-27T00:53:37Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Make foreground indexes not create a client
Branch: master
https://github.com/mongodb/mongo/commit/283fc0d291c196232cebc8b964178037aa71f943

Comment by auto [ 27/Dec/12 ]

Author:

{u'date': u'2012-12-27T00:59:22Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Use multiplatform sleep function
Branch: master
https://github.com/mongodb/mongo/commit/325ec6e4d66dfdc075b47a804314eec8f7ea3eb2

Comment by auto [ 27/Dec/12 ]

Author:

{u'date': u'2012-12-27T01:42:47Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Fix test to handle other tests not cleaning up
Branch: master
https://github.com/mongodb/mongo/commit/c5e5e6b9c0d9609234afd1390984d08f6055801b

Comment by auto [ 27/Dec/12 ]

Author:

{u'date': u'2012-12-27T02:23:22Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Fix compact index kill criteria
Branch: master
https://github.com/mongodb/mongo/commit/7962e49b40c2bf344187df4c6fdd857c2255d759

Comment by auto [ 28/Dec/12 ]

Author:

{u'date': u'2012-12-28T20:05:56Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Test fixes
Branch: master
https://github.com/mongodb/mongo/commit/e733331c7edb8871ba2dc58f20fbd2f40bf3adda

Comment by auto [ 02/Jan/13 ]

Author:

{u'date': u'2013-01-02T15:41:44Z', u'email': u'kristina@10gen.com', u'name': u'Kristina'}

Message: SERVER-2771 Revert command handling

This reverts commit e733331c7edb8871ba2dc58f20fbd2f40bf3adda.
This reverts commit ccf86a0fdbe7f9c9a45e8fad0410a7eef777fea2.
This reverts commit 7962e49b40c2bf344187df4c6fdd857c2255d759.
This reverts commit c5e5e6b9c0d9609234afd1390984d08f6055801b.
This reverts commit 325ec6e4d66dfdc075b47a804314eec8f7ea3eb2.
This reverts commit 283fc0d291c196232cebc8b964178037aa71f943.
This reverts commit 2f21426abe2d75487c91fb361154261969d303cd.
Branch: master
https://github.com/mongodb/mongo/commit/e04a7396ea753ea6dbff89028c11ed689868d707

Comment by Colin Howe [ 10/Jan/13 ]

We just got hit by this... all our slaves stopped responding to queries and our clients just hung (no timeouts). Looking forward to seeing a fix!

Comment by auto [ 25/Jan/13 ]

Author:

{u'date': u'2013-01-25T16:52:45Z', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-2771 --noIndexBuildRetry does not take a param
Branch: master
https://github.com/mongodb/mongo/commit/7a8fdfc75c26efc9aa4b3849587ef0995ffd9a36

Comment by Colin Howe [ 28/Mar/13 ]

Could we get an update on what is happening here? Some of the comments indicate that this was fixed in 2.2, some say 2.4 and then the status says 2.5.

We just got bitten by this (again) after thinking that it was fixed 2.2.3...

Pulling things out of replica sets to add indices really isn't a viable option. At least not for something that claims to make scaling easy with things like sharding support.

Comment by Eric Milkie [ 28/Mar/13 ]

The fixVersion field, version 2.5.0, is the current target for this. We have one more thing to iron out before releasing this feature.

Generated at Thu Jun 21 17:01:21 UTC 2018 using JIRA 7.8.2#78002-sha1:944b71ecbe2e09c23503821098ef280c785b44a8.