[SERVER-3160] replication initial sync should use the fast compact command code to build indexes Created: 27/May/11  Updated: 12/Jul/16  Resolved: 06/Dec/12

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.3.2

Type: Improvement Priority: Major - P3
Reporter: Dwight Merriman Assignee: Ben Becker
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-5040 Cloner can fail to create unique inde... Closed
is depended on by SERVER-7527 Improve speed of replication when in ... Closed
Duplicate
is duplicated by SERVER-2731 Optimize index building during initia... Closed
Related
is related to SERVER-7946 Command to build multiple indexes at ... Closed
Participants:

 Description   

The new compact command has been optimized to be very fast at building several indexes during the compaction of a collection. Keys for all indexes are written to temporary files during a single scan of the collection data.

The same technique could be applied to initial sync and would make it faster for big collections with several indexes. See the compact code - given that is now done it should be fairly easy to plug in to initial sync (clonecollection).



 Comments   
Comment by auto [ 07/Dec/12 ]

Author:

{u'date': u'2012-12-07T06:46:22Z', u'email': u'benjamin.becker@gmail.com', u'name': u'Ben Becker'}

Message: SERVER-3160: fix check for existing presorter
Branch: master
https://github.com/mongodb/mongo/commit/81f940e7d8b5429264136f77ee80ed4d3c8948fd

Comment by auto [ 06/Dec/12 ]

Author:

{u'date': u'2012-12-06T22:38:10Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-3160: Sort index keys during clone
Branch: master
https://github.com/mongodb/mongo/commit/5e5afe90d148fb93d4c0b05470681d57de5bc3e9

Comment by Eric Milkie [ 13/Nov/12 ]

Yes, I say add it to Cloner::go(). The code correctness should be pretty solid, as it's been in compact() since version 2.0.

Comment by Eric Milkie [ 13/Nov/12 ]

Interesting note about the global precalced variable.. with DB level locking, compact command only takes a DB write lock now. So is it possible for precalced to have collisions if two compacts are running at once?

With cloner, we may have the same issue, since cloner is used by more things than just initial sync – there are several user-facing commands that call into cloner.

I think the actual way to fix this is to move the initial setup logic for indexes out of DataFileMgr::Insert() and into its own function that we can call, and pass precalced as a parameter (or via a state object).
However, for the interim, we may just need to use a mutex here to protect the global.

Comment by Eliot Horowitz (Inactive) [ 07/Mar/12 ]

It should also get get ops up to date so as to be as efficient as possible.
Also fixes SERVER-5040

Generated at Thu Feb 08 03:02:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.