[DOCS-14391] Investigate changes in SERVER-56054: Change minThreads value for replication writer thread pool to 0 Created: 29/Apr/21  Updated: 13/Nov/23  Due: 11/Jun/21  Resolved: 14/Jun/21

Status: Closed
Project: Documentation
Component/s: manual, Server
Affects Version/s: None
Fix Version/s: 4.2.15, 4.4.7, 5.0.0-rc0, 4.0.26, Server_Docs_20231030, Server_Docs_20231106, Server_Docs_20231105, Server_Docs_20231113

Type: Task Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Andrew Feierabend (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
backported by DOCS-14473 [BACKPORT] [v4.4] Change minThreads v... Closed
backported by DOCS-14546 [BACKPORT] [v4.2] Server: Change minT... Closed
backported by DOCS-14596 [SERVER] [BACKPORT v4.0] Change minTh... Closed
Documented
documents SERVER-56054 Change minThreads value for replicati... Closed
Participants:
Days since reply: 2 years, 24 weeks, 6 days ago
Epic Link: DOCSP-9747
Story Points: 3

 Description   

Description

Downstream Change Summary

Added server parameter replWriterMinThreadCount - The minimum number of threads in the thread pool used to apply the oplog with default 0. Secondary oplog application will use up to replWriterThreadCount threads (https://docs.mongodb.com/manual/reference/parameters/#mongodb-parameter-param.replWriterThreadCount). If the newly added replWriterMinThreadCount is less than replWriterThreadCount, the thread pool will timeout idle threads and only keep replWriterMinThreadCount idle threads in the pool. (replWriterMinThreadCount must be less than or equal to replWriterThreadCount)

Description of Linked Ticket

SERVER-54805 describes a case where the replication machinery on replica set secondaries ceases to make progress, with the symptom being that all threads in the replication writer thread pool are idle and the thread driving secondary replication is simultaneously blocked waiting for those writer threads to finish their work.

So far, this behavior has only manifest on systems with glibc versions susceptible to this glibc pthread condition variable bug. While I have not been able to build a minimal reproduction using our ThreadPool type, the scenario proven to exist in this blog post about using TLA+ to model glibc condition variables is perfectly analogous to how replication uses thread pools. In this scenario, a signal delivery that is lost due to the glibc bug leads to incomplete work being left in the thread pool, and no threads waking up to perform the work.

Fortunately, a low-risk workaround for this bug as it manifests in the replication system's use of ThreadPool exists. By setting minThreads to 0 instead of its current value, which is equal to maxThreads, we ensure that all waits performed by worker threads eventually wake up due to expiration of the idle thread timeout.

The task in this ticket is to change the value of minThreads in the writer thread pool used by replication to 0. This will not eliminate all possible failures due to the glibc bug, but it will eliminate the only one we've seen in practice until such time as the bug in glibc is corrected.

Scope of changes

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)



 Comments   
Comment by Githook User [ 19/Aug/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: v4.0
https://github.com/mongodb/docs/commit/63d93cbd28f6fa31c75d7e50faafa105dcf8ecd1

Comment by Githook User [ 23/Jul/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: v4.2
https://github.com/mongodb/docs/commit/46b6ed8dd603becbeb2b1fc0bfb34167e5daedd7

Comment by Githook User [ 16/Jul/21 ]

Author:

{'name': 'andf-mongodb', 'email': '55111344+andf-mongodb@users.noreply.github.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 Fix added version for v4.4.7
Branch: v4.4
https://github.com/mongodb/docs/commit/0ec3922ee509fa71d522614ad9059c42f1a6cec5

Comment by Githook User [ 16/Jul/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: v4.4
https://github.com/mongodb/docs/commit/c8e2a91ff4530648fc43ae919de8c374ff010818

Comment by Githook User [ 14/Jun/21 ]

Author:

{'name': 'andf-mongodb', 'email': '55111344+andf-mongodb@users.noreply.github.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 Fix added version for v4.4.7
Branch: v4.4.7
https://github.com/mongodb/docs/commit/0f7bae20df8530e2a70e69aac4db53b25ba85d16

Comment by Githook User [ 14/Jun/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: v4.2.15
https://github.com/mongodb/docs/commit/274a75b5e04ab6e907e41424c9a446d80606d5b2

Comment by Githook User [ 14/Jun/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: v4.4.7
https://github.com/mongodb/docs/commit/63f1d33559a0136e0237c15580ffd8c397c6cfb3

Comment by Githook User [ 14/Jun/21 ]

Author:

{'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}

Message: DOCS-14391 add replWriterMinThreadCount param
Branch: master
https://github.com/mongodb/docs/commit/b28a119ec97f523be8a5468d469fed522da6efad

Generated at Thu Feb 08 08:10:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.