-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Atlas Streams
-
Sprint 69
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
In the ChangeStreamSourceOperator, today, we end up issuing a getMore (via the CXX driver's change_stream/iterator abstractions) and then we loop over the retrieved events, extracting events and their resume tokens and then pushing them into internal batches/queues. After we iterate to the end, we end up issuing another getMore call.
If we end up pulling in a large number of events from the server, then we are not efficiently interleaving the network getMore calls with the above loop. We can instead try to write the code in a such a way that the getMore call is issued immediately and then iterate over the currently received events.
This may require us to use the C API directly.