[SERVER-54939] Investigate secondary batching behavior in v4.4 Created: 04/Mar/21  Updated: 19/Dec/22  Resolved: 13/Jun/22

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Jason Chan Assignee: Matthew Russotto
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File majority writes v4.4.png     PNG File oplogFetcherUsesExhaust=false.png     PNG File v4.2.png     PNG File v4.4.png     PNG File w-majority-8-threads.png    
Issue Links:
Backports
Depends
is depended on by SERVER-53667 High rate of journal flushes on secon... Closed
Documented
is documented by DOCS-14315 Investigate changes in SERVER-54939: ... Backlog
Duplicate
is duplicated by SERVER-53667 High rate of journal flushes on secon... Closed
Related
related to SERVER-55606 Majority writes with j: false perform... Open
related to SERVER-71503 Big disk IO (write) regression (×8) w... Closed
related to SERVER-65723 Add tunable parameter to improve batc... Closed
is related to SERVER-48522 Regression after mass deletion Open
is related to SERVER-59776 50% regression in single multi-update Closed
Backport Requested:
v5.0
Sprint: Repl 2021-03-22, Repl 2021-04-05, Repl 2022-03-21, Repl 2022-04-04, Repl 2022-04-18, Repl 2022-05-02, Repl 2022-05-16, Repl 2022-05-30, Repl 2022-06-13, Repl 2022-06-27
Participants:
Case:

 Description   

SERVER-53667 reported that 4.4 secondaries were seen to be applying oplogs in a much larger number of batches than seen in 4.2. We should see whether there was a regression in batch behavior from 4.2 to 4.4.



 Comments   
Comment by Matthew Russotto [ 13/Jun/22 ]

The reduced batch size on 4.4 is due to changes designed to improve read concern majority latency. We have added a tunable parameter in SERVER-65723 to allow increased batching at the cost of higher read concern majority latency.

Comment by Judah Schvimer [ 24/Mar/21 ]

In order for writeConcern 'majority' to provide its durability, journaling is required. Depending on the value of writeConcernMajorityJournalDefault, writeConcern 'majority' will journal by default, and will not journal if j:false or that flag (set to false) are provided (with some quirks with the inMemory storage engine).

Comment by Dianna Hohensee (Inactive) [ 24/Mar/21 ]

If I understand 'majority' write concern correctly, it doesn't implicitly require {j:true}. Therefore, it would make a lot of sense to me to use your proposal of a {requiresFlush: true} flag whenever the user specifies {j:true}, and otherwise let journal flushing occur on a periodic basis.

I don't remember how we determine majority committed, whether it currently depends upon durability as well as majority. This might have to change, if we're doing stronger guarantees than necessary. We would also need to survey, perhaps, what operations need persistence vs what ops don't; or alternatively, make all internal ops attach {requiresFlush: true} initially, and remove them later on on an individual basis. I think the latter would be safer, as I wouldn't be surprised if there are a lot of subtleties around this sort of thing, but I don't know.

Comment by Russell Andolina (Inactive) [ 15/Mar/21 ]

lingzhi.deng - do we have an approximate timeframe on this fix? Bell is saying they'll hold off on their planned upgrade until we have resolution.

Comment by Eric Lafontaine [ 08/Mar/21 ]

Hi,

I will be watching this issue along with SERVER-53667, I require a fix ASAP.  Some production workload is running with this version as we weren't observing that behaviour in Dev.

Regards,

Eric Lafontaine

Generated at Thu Feb 08 05:34:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.