[SERVER-81063] Streams: Async DLQ inserts with batching Created: 14/Sep/23 Updated: 29/Oct/23 Resolved: 25/Sep/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.2.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Aadesh Patel (Inactive) | Assignee: | Aadesh Patel (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Atlas Streams
|
| Backwards Compatibility: | Fully Compatible |
| Sprint: | Sprint 32 |
| Participants: |
| Description |
|
Right now we synchronously insert one document into the DLQ at a time, we should move the i/o work to a background thread and make all inserts into the DLQ async. The worker thread should push all documents destined for the DLQ into a queue that the DLQ background thread is constantly consuming from. Ideally the work we do here can be reused to do something similar in the merge operator. Slack context: https://mongodb.slack.com/archives/C05P740L0VC/p1694540160003889 |