Better interleaving of getMore with retrieved events, change stream source benchmark

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • Sprint 69
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      In the ChangeStreamSourceOperator, today, we end up issuing a getMore (via the CXX driver's change_stream/iterator abstractions) and then we loop over the retrieved events, extracting events and their resume tokens and then pushing them into internal batches/queues. After we iterate to the end, we end up issuing another getMore call.

      If we end up pulling in a large number of events from the server, then we are not efficiently interleaving the network getMore calls with the above loop. We can instead try to write the code in a such a way that the getMore call is issued immediately and then iterate over the currently received events.

      This may require us to use the C API directly.

            Assignee:
            Andrea Rahmanan
            Reporter:
            Mayuresh Kulkarni
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: