Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-74501

Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 7.0.0-rc0, 6.3.0-rc1, 5.0.16, 6.0.6
    • Affects Version/s: 5.0.15, 6.3.0-rc0
    • Component/s: None
    • Fully Compatible
    • ALL
    • v6.3, v6.0, v5.0

      Doesn't impact clusters that have the _SecondaryThrottle turned off or have the balancer off.

      MigrationBatchFetcher's  destructor depends on calling inserterWorkerThreadPool shutdown/join to drain all the scheduled jobs and join on the threadPool spawns another extra thread to drain it.

      Also, secondaryThrottle is disabled for a chunkMigrationConcurrency of more than 1 because of how sessions are checked out/in. 

      When the MigrationBatchFetcher calls join on the inserterThreadPool and there are more jobs to process in the pool, it can spawn an extra thread to drain the pool. So now, although the threadCnt is 1 here, 2 threads are competing for secondaryThrottle and the invariant is caused.

       

      Invariant failure string in the log file looked like - 

      {"t":\{"$date":"2023-02-28T19:58:58.693+00:00"}

      ,"s":"F", "c":"-", "id":23079, "ctx":"ChunkMigrationInserters-1","msg":"Invariant failure","attr":{"expr":"checkedOutSession","file":"src/mongo/db/session_catalog.cpp","line":302}}

       

      Workaround

      Either one of the following would work

      1. Turn the balancer off

      2. Turn the secondaryThrottle off(with the balancer still running).

       

            Assignee:
            randolph@mongodb.com Randolph Tan
            Reporter:
            rachita.dhawan@mongodb.com Rachita Dhawan
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: