Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6883

index creation on secondaries need not block readers

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: Indexing, Replication
    • Labels:
      None

      Description

      On a secondary, a "foreground" (normal) index build could be done while still letting readers read. then we just swap in the index when it is ready with a very short write lock. that is, assuming here there are no other writers than the replication sync thread.

      while the index build is in progress i'm assuming other ops would not be applied, so some lag would accumulate, but it would be nice to still be readable. thoughts?

        Issue Links

          Activity

          Hide
          milkie Eric Milkie added a comment -

          This sounds like it would just be a background index build, and we'd have the replication thread wait for it to be done before it continued on with replication. That might be easier to implement than a new hybrid foreground index build that doesn't take a write lock until at the very end.

          Show
          milkie Eric Milkie added a comment - This sounds like it would just be a background index build, and we'd have the replication thread wait for it to be done before it continued on with replication. That might be easier to implement than a new hybrid foreground index build that doesn't take a write lock until at the very end.
          Hide
          dwight_10gen Dwight Merriman added a comment -

          @eric i am assuming this would be the fg indexing algorithm which is bottom up with external disk sort and thus can build large indexes efficiently.

          Show
          dwight_10gen Dwight Merriman added a comment - @eric i am assuming this would be the fg indexing algorithm which is bottom up with external disk sort and thus can build large indexes efficiently.
          Hide
          dwight_10gen Dwight Merriman added a comment -

          eric i wonder with the legacy storage engine if this would be an alternate (better) way to build the indexes. so maybe there is some unification.

          Show
          dwight_10gen Dwight Merriman added a comment - eric i wonder with the legacy storage engine if this would be an alternate (better) way to build the indexes. so maybe there is some unification.
          Hide
          dwight_10gen Dwight Merriman added a comment -

          with multiple storage engines, the question is does this make sense. i think 2 yrs ago when posted i was thinking of reading from the non-private mmap.

          however, the general notion could still make sense. we could say "build index x" to storage engine whatever; it then says "inprog" and we then keep allowing reads (only – otherwise it turns into bg indexing complexity). that is to say, any storage engine probably could in theory be making an index and everything else is still readable, as it is unlikely anything else mutates during that operation until it is done.

          we could give engine option to return "inprog" vs blocking until done or something and leave it up to the implementor?

          i suppose with enough concurrency the issue (at least with the serialness described above) goes away. but it is pretty typical for storage engines to block a collection C (only) when you make an index on C. (some don't but many/most do)

          Show
          dwight_10gen Dwight Merriman added a comment - with multiple storage engines, the question is does this make sense. i think 2 yrs ago when posted i was thinking of reading from the non-private mmap. however, the general notion could still make sense. we could say "build index x" to storage engine whatever; it then says "inprog" and we then keep allowing reads (only – otherwise it turns into bg indexing complexity). that is to say, any storage engine probably could in theory be making an index and everything else is still readable, as it is unlikely anything else mutates during that operation until it is done. we could give engine option to return "inprog" vs blocking until done or something and leave it up to the implementor? i suppose with enough concurrency the issue (at least with the serialness described above) goes away. but it is pretty typical for storage engines to block a collection C (only) when you make an index on C. (some don't but many/most do)

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: