[SERVER-6883] index creation on secondaries need not block readers Created: 28/Aug/12 Updated: 06/Dec/22 Resolved: 04/Mar/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance, Replication |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dwight Merriman | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
On a secondary, a "foreground" (normal) index build could be done while still letting readers read. then we just swap in the index when it is ready with a very short write lock. that is, assuming here there are no other writers than the replication sync thread. while the index build is in progress i'm assuming other ops would not be applied, so some lag would accumulate, but it would be nice to still be readable. thoughts? |
| Comments |
| Comment by Sara Williamson [ 04/Mar/19 ] |
|
Closing this ticket as the Hybrid Index Build project has removed foreground index builds. |
| Comment by Dwight Merriman [ 19/Sep/14 ] |
|
with multiple storage engines, the question is does this make sense. i think 2 yrs ago when posted i was thinking of reading from the non-private mmap. however, the general notion could still make sense. we could say "build index x" to storage engine whatever; it then says "inprog" and we then keep allowing reads (only – otherwise it turns into bg indexing complexity). that is to say, any storage engine probably could in theory be making an index and everything else is still readable, as it is unlikely anything else mutates during that operation until it is done. we could give engine option to return "inprog" vs blocking until done or something and leave it up to the implementor? i suppose with enough concurrency the issue (at least with the serialness described above) goes away. but it is pretty typical for storage engines to block a collection C (only) when you make an index on C. (some don't but many/most do) |
| Comment by Dwight Merriman [ 19/Sep/14 ] |
|
eric i wonder with the legacy storage engine if this would be an alternate (better) way to build the indexes. so maybe there is some unification. |
| Comment by Dwight Merriman [ 31/Aug/12 ] |
|
@eric i am assuming this would be the fg indexing algorithm which is bottom up with external disk sort and thus can build large indexes efficiently. |
| Comment by Eric Milkie [ 29/Aug/12 ] |
|
This sounds like it would just be a background index build, and we'd have the replication thread wait for it to be done before it continued on with replication. That might be easier to implement than a new hybrid foreground index build that doesn't take a write lock until at the very end. |