[SERVER-3160] replication initial sync should use the fast compact command code to build indexes Created: 27/May/11 Updated: 12/Jul/16 Resolved: 06/Dec/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | 2.3.2 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dwight Merriman | Assignee: | Ben Becker |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
The new compact command has been optimized to be very fast at building several indexes during the compaction of a collection. Keys for all indexes are written to temporary files during a single scan of the collection data. The same technique could be applied to initial sync and would make it faster for big collections with several indexes. See the compact code - given that is now done it should be fairly easy to plug in to initial sync (clonecollection). |
| Comments |
| Comment by auto [ 07/Dec/12 ] |
|
Author: {u'date': u'2012-12-07T06:46:22Z', u'email': u'benjamin.becker@gmail.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 06/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T22:38:10Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by Eric Milkie [ 13/Nov/12 ] |
|
Yes, I say add it to Cloner::go(). The code correctness should be pretty solid, as it's been in compact() since version 2.0. |
| Comment by Eric Milkie [ 13/Nov/12 ] |
|
Interesting note about the global precalced variable.. with DB level locking, compact command only takes a DB write lock now. So is it possible for precalced to have collisions if two compacts are running at once? With cloner, we may have the same issue, since cloner is used by more things than just initial sync – there are several user-facing commands that call into cloner. I think the actual way to fix this is to move the initial setup logic for indexes out of DataFileMgr::Insert() and into its own function that we can call, and pass precalced as a parameter (or via a state object). |
| Comment by Eliot Horowitz (Inactive) [ 07/Mar/12 ] |
|
It should also get get ops up to date so as to be as efficient as possible. |