[SERVER-78645] Improve kafka $emit Created: 03/Jul/23  Updated: 29/Oct/23  Resolved: 03/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Mihai Andrei Assignee: Mihai Andrei
Resolution: Fixed Votes: 0
Labels: init-337-m2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Atlas Streams
Backwards Compatibility: Fully Compatible
Sprint: Sprint 28
Participants:

 Description   

Some ideas:

  • Use a multithreaded approach similar to some of the $source stages (namely, kafka and change streams) where a separate thread is responsible for flushing documents (in this case, to our kafka topic)
  • Allow for more configuration options


 Comments   
Comment by Mihai Andrei [ 26/Jul/23 ]

Moving this back to 'open' as I'm planning on opening a PR to implement multithreading for this operator.

Comment by Mihai Andrei [ 06/Jul/23 ]

At a minimum, this should implement using a multithreaded approach (i.e. have a thread responsible for calling 'Producer :: produce()). The rest of the suggested improvements:

src/streams/exec/kafka_emit_operator.cpp:    // TODO SERVER-78645: Set more config options that could be useful.
src/streams/exec/kafka_emit_operator.h:        // TODO SERVER-78645: Consider adding more configuration options.
src/streams/exec/kafka_emit_operator.h:    // TODO SERVER-78645: Determine if this can be deduplicated with

Are 'nice to haves'

Generated at Thu Feb 08 06:38:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.