[SERVER-74501] Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread Created: 01/Mar/23  Updated: 29/Oct/23  Resolved: 20/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.15, 6.3.0-rc0
Fix Version/s: 7.0.0-rc0, 6.3.0-rc1, 5.0.16, 6.0.6

Type: Bug Priority: Critical - P2
Reporter: Rachita Dhawan Assignee: Randolph Tan
Resolution: Fixed Votes: 0
Labels: RDY
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.3, v6.0, v5.0
Participants:
Case:

 Description   

Doesn't impact clusters that have the _SecondaryThrottle turned off or have the balancer off.

MigrationBatchFetcher's  destructor depends on calling inserterWorkerThreadPool shutdown/join to drain all the scheduled jobs and join on the threadPool spawns another extra thread to drain it.

Also, secondaryThrottle is disabled for a chunkMigrationConcurrency of more than 1 because of how sessions are checked out/in. 

When the MigrationBatchFetcher calls join on the inserterThreadPool and there are more jobs to process in the pool, it can spawn an extra thread to drain the pool. So now, although the threadCnt is 1 here, 2 threads are competing for secondaryThrottle and the invariant is caused.

 

Invariant failure string in the log file looked like - 

{"t":\{"$date":"2023-02-28T19:58:58.693+00:00"}

,"s":"F", "c":"-", "id":23079, "ctx":"ChunkMigrationInserters-1","msg":"Invariant failure","attr":{"expr":"checkedOutSession","file":"src/mongo/db/session_catalog.cpp","line":302}}

 

Workaround

Either one of the following would work

1. Turn the balancer off

2. Turn the secondaryThrottle off(with the balancer still running).

 



 Comments   
Comment by Githook User [ 20/Mar/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread

(cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20)
Branch: v6.0
https://github.com/mongodb/mongo/commit/033418d940372570b12ec4233946e9675fc05226

Comment by Githook User [ 07/Mar/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread

(cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20)
Branch: v5.0
https://github.com/mongodb/mongo/commit/84a03cb20852a2dc0247481f7da7a023688bd146

Comment by Githook User [ 06/Mar/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread

(cherry picked from commit 00dc7293c5d815e2104b5fe0c9a225332af3dfec)
Branch: v6.3
https://github.com/mongodb/mongo/commit/bf4b3ae9b3eb5c376201135155ef6738bfef0e20

Comment by Githook User [ 06/Mar/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread
Branch: master
https://github.com/mongodb/mongo/commit/00dc7293c5d815e2104b5fe0c9a225332af3dfec

Generated at Thu Feb 08 06:27:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.