Streams: Async DLQ inserts with batching

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 7.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • Fully Compatible
    • Sprint 32
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Right now we synchronously insert one document into the DLQ at a time, we should move the i/o work to a background thread and make all inserts into the DLQ async. The worker thread should push all documents destined for the DLQ into a queue that the DLQ background thread is constantly consuming from.

      Ideally the work we do here can be reused to do something similar in the merge operator.

      Slack context: https://mongodb.slack.com/archives/C05P740L0VC/p1694540160003889

            Assignee:
            Aadesh Patel (Inactive)
            Reporter:
            Aadesh Patel (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: