[SERVER-74501] Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread Created: 01/Mar/23 Updated: 29/Oct/23 Resolved: 20/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 5.0.15, 6.3.0-rc0 |
| Fix Version/s: | 7.0.0-rc0, 6.3.0-rc1, 5.0.16, 6.0.6 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Rachita Dhawan | Assignee: | Randolph Tan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | RDY | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Requested: |
v6.3, v6.0, v5.0
|
||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
Doesn't impact clusters that have the _SecondaryThrottle turned off or have the balancer off. MigrationBatchFetcher's destructor depends on calling inserterWorkerThreadPool shutdown/join to drain all the scheduled jobs and join on the threadPool spawns another extra thread to drain it. Also, secondaryThrottle is disabled for a chunkMigrationConcurrency of more than 1 because of how sessions are checked out/in. When the MigrationBatchFetcher calls join on the inserterThreadPool and there are more jobs to process in the pool, it can spawn an extra thread to drain the pool. So now, although the threadCnt is 1 here, 2 threads are competing for secondaryThrottle and the invariant is caused.
Invariant failure string in the log file looked like -
Workaround Either one of the following would work 1. Turn the balancer off 2. Turn the secondaryThrottle off(with the balancer still running).
|
| Comments |
| Comment by Githook User [ 20/Mar/23 ] |
|
Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}Message: (cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20) |
| Comment by Githook User [ 07/Mar/23 ] |
|
Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}Message: (cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20) |
| Comment by Githook User [ 06/Mar/23 ] |
|
Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}Message: (cherry picked from commit 00dc7293c5d815e2104b5fe0c9a225332af3dfec) |
| Comment by Githook User [ 06/Mar/23 ] |
|
Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}Message: |