[SERVER-6883] index creation on secondaries need not block readers Created: 28/Aug/12  Updated: 06/Dec/22  Resolved: 04/Mar/19

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Dwight Merriman Assignee: Backlog - Storage Execution Team
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-25168 Foreground index build blocks all R/W... Closed
Related
related to SERVER-20328 Allow secondary reads while applying ... Closed
is related to SERVER-6836 Allow index builds on secondaries wit... Closed
is related to SERVER-2771 Background index builds on replica se... Closed
is related to SERVER-18301 Allow reads during foreground index b... Closed
Assigned Teams:
Storage Execution
Participants:

 Description   

On a secondary, a "foreground" (normal) index build could be done while still letting readers read. then we just swap in the index when it is ready with a very short write lock. that is, assuming here there are no other writers than the replication sync thread.

while the index build is in progress i'm assuming other ops would not be applied, so some lag would accumulate, but it would be nice to still be readable. thoughts?



 Comments   
Comment by Sara Williamson [ 04/Mar/19 ]

Closing this ticket as the Hybrid Index Build project has removed foreground index builds.

Comment by Dwight Merriman [ 19/Sep/14 ]

with multiple storage engines, the question is does this make sense. i think 2 yrs ago when posted i was thinking of reading from the non-private mmap.

however, the general notion could still make sense. we could say "build index x" to storage engine whatever; it then says "inprog" and we then keep allowing reads (only – otherwise it turns into bg indexing complexity). that is to say, any storage engine probably could in theory be making an index and everything else is still readable, as it is unlikely anything else mutates during that operation until it is done.

we could give engine option to return "inprog" vs blocking until done or something and leave it up to the implementor?

i suppose with enough concurrency the issue (at least with the serialness described above) goes away. but it is pretty typical for storage engines to block a collection C (only) when you make an index on C. (some don't but many/most do)

Comment by Dwight Merriman [ 19/Sep/14 ]

eric i wonder with the legacy storage engine if this would be an alternate (better) way to build the indexes. so maybe there is some unification.

Comment by Dwight Merriman [ 31/Aug/12 ]

@eric i am assuming this would be the fg indexing algorithm which is bottom up with external disk sort and thus can build large indexes efficiently.

Comment by Eric Milkie [ 29/Aug/12 ]

This sounds like it would just be a background index build, and we'd have the replication thread wait for it to be done before it continued on with replication. That might be easier to implement than a new hybrid foreground index build that doesn't take a write lock until at the very end.

Generated at Thu Feb 08 03:13:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.