Move BucketCatalog committer assignment to commit time

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Execution Team 2021-03-22
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Currently the decision of which writer commits a bucket is made when an individual writer makes the first uncommitted insert to a bucket. In cases with large batches and many writers, this can be highly problematic. A single writer W1 may have committer status on multiple buckets, e.g. B1 and B2. Many writers may be waiting on B2 while W1 is busy committing B1. One of these writers, say W2, could be committing B2, increasing throughput.

      Shifting the committer assignment to commit time rather than insert time should allow us to minimize wait time.

      At the same time, we will also simplify the state management a bit. Instead of keeping a promise per measurement insert, we will maintain likely a single state variable (promise or condition variable, etc.) at a time, or maybe a few, but definitely fewer than we currently maintain.

            Assignee:
            Dan Larkin-York
            Reporter:
            Dan Larkin-York
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: