Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-91318

CatalogCache refresh can deadlock on the config shard

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.1.0-rc0, 8.0.0-rc10
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • Fully Compatible
    • ALL
    • v8.0
    • CAR Team 2024-06-24
    • 0

      The CatalogCache executor pool is limited to 6 threads. CollectionCache, DatabaseCache, and IndexCache share this executor.

      1. In the config shard, 6 threads are refreshing the CollectionCache for different namespaces, occupying all threads in the pool.
      2. The refresh eventually reaches out to the config server through ConfigServerCatalogCacheLoader, and runs an aggregation on it.
      3. The aggregation uses (readPreference: nearest).
      4. The aggregation doesn't use timeout when the node itself is config server.
      5. Targeting returns this node.
      6. Execution of the aggregation, which contains a $unionWith, and preparing the subpipeline involves routing.
      7. This schedules a job in the CatalogCache thread pool, to fetch the routing information for "config" database.
      8. And waits for it to complete, which never does because no threads in the pool will ever be freed due to the circular dependency.

       

            Assignee:
            yujin.kang@mongodb.com Yujin Kang Park
            Reporter:
            yujin.kang@mongodb.com Yujin Kang Park
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: