[DOCS-14391] Investigate changes in SERVER-56054: Change minThreads value for replication writer thread pool to 0 Created: 29/Apr/21 Updated: 13/Nov/23 Due: 11/Jun/21 Resolved: 14/Jun/21 |
|
| Status: | Closed |
| Project: | Documentation |
| Component/s: | manual, Server |
| Affects Version/s: | None |
| Fix Version/s: | 4.2.15, 4.4.7, 5.0.0-rc0, 4.0.26, Server_Docs_20231030, Server_Docs_20231106, Server_Docs_20231105, Server_Docs_20231113 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Backlog - Core Eng Program Management Team | Assignee: | Andrew Feierabend (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Days since reply: | 2 years, 24 weeks, 6 days ago | ||||||||||||||||||||||||
| Epic Link: | DOCSP-9747 | ||||||||||||||||||||||||
| Story Points: | 3 | ||||||||||||||||||||||||
| Description |
DescriptionDownstream Change Summary Added server parameter replWriterMinThreadCount - The minimum number of threads in the thread pool used to apply the oplog with default 0. Secondary oplog application will use up to replWriterThreadCount threads (https://docs.mongodb.com/manual/reference/parameters/#mongodb-parameter-param.replWriterThreadCount). If the newly added replWriterMinThreadCount is less than replWriterThreadCount, the thread pool will timeout idle threads and only keep replWriterMinThreadCount idle threads in the pool. (replWriterMinThreadCount must be less than or equal to replWriterThreadCount) Description of Linked Ticket So far, this behavior has only manifest on systems with glibc versions susceptible to this glibc pthread condition variable bug. While I have not been able to build a minimal reproduction using our ThreadPool type, the scenario proven to exist in this blog post about using TLA+ to model glibc condition variables is perfectly analogous to how replication uses thread pools. In this scenario, a signal delivery that is lost due to the glibc bug leads to incomplete work being left in the thread pool, and no threads waking up to perform the work. Fortunately, a low-risk workaround for this bug as it manifests in the replication system's use of ThreadPool exists. By setting minThreads to 0 instead of its current value, which is equal to maxThreads, we ensure that all waits performed by worker threads eventually wake up due to expiration of the idle thread timeout. The task in this ticket is to change the value of minThreads in the writer thread pool used by replication to 0. This will not eliminate all possible failures due to the glibc bug, but it will eliminate the only one we've seen in practice until such time as the bug in glibc is corrected. Scope of changesImpact to Other DocsMVP (Work and Date)Resources (Scope or Design Docs, Invision, etc.) |
| Comments |
| Comment by Githook User [ 19/Aug/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 23/Jul/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 16/Jul/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': '55111344+andf-mongodb@users.noreply.github.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 16/Jul/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 14/Jun/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': '55111344+andf-mongodb@users.noreply.github.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 14/Jun/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 14/Jun/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |
| Comment by Githook User [ 14/Jun/21 ] |
|
Author: {'name': 'andf-mongodb', 'email': 'andrew.feierabend@mongodb.com', 'username': 'andf-mongodb'}Message: |