[SERVER-69959] Introduce majority committed point advancement notification mechanism Created: 26/Sep/22  Updated: 29/Oct/23  Resolved: 29/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Denis Grebennicov Assignee: Denis Grebennicov
Resolution: Fixed Votes: 0
Labels: pm-2334-followup
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Related
related to SERVER-74526 Change stream opened against a second... Closed
related to SERVER-74580 Complete TODO listed in SERVER-69959 Closed
related to SERVER-71161 Investigate removal of `signalOplogWa... Closed
related to SERVER-74555 Re-introduce majority commit point ad... Closed
is related to SERVER-66840 Investigate and fix testcases Closed
Backwards Compatibility: Fully Compatible
Sprint: QE 2022-10-17, QE 2022-10-31, QE 2022-11-14, QE 2022-11-28, QE 2022-12-12
Participants:
Linked BF Score: 5

 Description   

While working on SERVER-66840 we discovered the issue with the test jstests/change_streams/only_wake_getmore_for_relevant_changes.js
 
The issue is that it was agreed that change collections are majority committed by design. In this case the regular insert_listeners do not work with change collections as desired, as they wake the listeners based on local writes. This results in redundant wakes and no documents being returned to the client, as when local write is performed, the write is not guaranteed to be majority committed just yet.
Oplog works around this issue by having extra signaling mechanism that wakes up upon timestamp advancement in oplog.
 
The decided solution to this issue is to make tailable awaitable cursors on collections (not only change collection or oplog) that have read concern majority to wait on majority point advacement notification instead of waiting for the local writes provided by the capped notifier.

louis.williams@mongodb.com has implemented a prototype to solve a similar problem https://evergreen.mongodb.com/filediff/62e15eaf7742ae1d62082175/?patch_number=0



 Comments   
Comment by Kyle Suarez [ 06/Mar/23 ]

Note: this change was reverted by SERVER-74526; it will be reintroduced in SERVER-74555.

Comment by Githook User [ 03/Mar/23 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-74526 Fix high CPU utilization for a change streams opened against secondary nodes

This reverts commit 34ac49477b87e183637f68cda828ecff8b393c64. Future
work is needed to reintroduce the changes from SERVER-69959 without
causing the problematic "busy wait" behavior on secondary nodes.
Branch: master
https://github.com/mongodb/mongo/commit/67773ced2cb8c57935b9ea08331fe69bbc608d57

Comment by Githook User [ 02/Mar/23 ]

Author:

{'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}

Message: SERVER-74526 Fix high CPU utilization for a change streams opened against secondary nodes

This reverts commit 34ac49477b87e183637f68cda828ecff8b393c64. Future
work is needed to reintroduce the changes from SERVER-69959 without
causing the problematic "busy wait" behavior on secondary nodes.
Branch: v6.3
https://github.com/mongodb/mongo/commit/96763fa1fef7faa2513afd2618d4d039ee70a6fe

Comment by Githook User [ 29/Nov/22 ]

Author:

{'name': 'Denis Grebennicov', 'email': 'denis.grebennicov@mongodb.com', 'username': 'denis631'}

Message: SERVER-69959 Introduce majority committed point advancement notification mechanism
Branch: master
https://github.com/mongodb/mongo/commit/34ac49477b87e183637f68cda828ecff8b393c64

Generated at Thu Feb 08 06:14:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.