[SERVER-84146] Save/Restore of checkpoint should account for duplicated documents Created: 13/Dec/23  Updated: 06/Feb/24

Status: Needs Scheduling
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Mayuresh Kulkarni Assignee: Backlog - Atlas Streams
Resolution: Unresolved Votes: 0
Labels: init-337-ga
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Atlas Streams
Participants:

 Description   

Consider a pipeline with a `hoppingWindow`, like this one:

 

pipeline: [{
$group:
{ _id:"$customerId", customerDocs: {$push:"$$ROOT"},
}
}]

Say there are 200 open windows. Now, an document will be absorbed into all these open windows. And, though the logical state size is O(200docs), the actual memory usage will still just be 1 doc since documents are cheaply copyable via ref-counting etc

Now, when such a state is checkpointed and recovered, we lose this sharing info and so today we will end up with 200 different docs after the recovery.

This causes ballooning in memory usage after the checkpoint has been recovered.


Generated at Thu Feb 08 06:54:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.