-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Distributed Query Execution
-
Query Execution
-
Fully Compatible
-
QE 2025-05-26, QE 2025-06-09, QE 2025-06-23
-
None
-
3
-
TBD
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
0
The AsyncResultsMerger has the responsibility of adding additional transaction participants to the currently running transaction whenever it receives a response from a shard that includes additional transaction participants in its metadata.
The AsyncResultsMerger currently runs its callback function that handles the responses to an asynchronous getMore request in the network thread. From there it is not safe to access the TransactionRouter and add the additional transaction participants, as another thread may be working on the same query and thus the same TransactionRouter instance concurrently.
The AsyncResultsMerger therefore unconditionally buffers the full received responses in the callback function that runs in the network thread.
These buffered responses are kept in memory until the AsyncResultsMerger is detached from the OperationContext or until when it consumes the next buffered document. In these cases it can safely access the TransactionRouter instance without concurrenty issues.
Buffering the full responses potentially leads to significant memory overhead in the AsyncResultsMerger, provided that we can run many concurrent queries and they all need to buffer the full responses until they can be processed.
As the TransactionRouter cannot be made thread-safe easily, an alternative solution is to make the AsyncResultsMerger extract the information about the additional transaction participants from the response metadata, and buffer only this metadata instead of the full responses. The transaction participants metadata should be much smaller than the full responses.
This should lead to significant memory savings in the AsyncResultsMerger.
- causes
-
SERVER-106098 Remove implicit coupling between async_transaction_rpc and transaction_router
-
- Backlog
-
-
SERVER-105981 Track, expose and test memory usage in AsyncResultsMerger
-
- Needs Scheduling
-
- is related to
-
SERVER-105384 Fix unsafe access to TransactionRouter in AsyncResultsMerger
-
- Closed
-
-
SERVER-105101 Make conditions for buffering and consuming full remote responses symmetric
-
- Closed
-
- related to
-
SERVER-105981 Track, expose and test memory usage in AsyncResultsMerger
-
- Needs Scheduling
-
-
SERVER-105982 Add system tests for mongos memory usage
-
- Needs Scheduling
-