Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-71627

Refreshed cached collection route info will severely block all client request when a cluster with 1 million chunks

XMLWordPrintableJSON

    • Sharding EMEA
    • Fully Compatible
    • v7.0, v6.0, v5.0, v4.4, v4.2
    • Sharding EMEA 2023-01-09, Sharding EMEA 2023-01-23, Sharding EMEA 2023-02-06, Sharding EMEA 2023-02-20, Sharding EMEA 2023-03-06, Sharding EMEA 2023-03-20, Sharding EMEA 2023-04-03, Sharding EMEA 2023-04-17, Sharding EMEA 2023-05-01, Sharding EMEA 2023-05-15, Sharding EMEA 2023-05-29, Sharding EMEA 2023-06-12, Sharding EMEA 2023-06-26, Sharding EMEA 2023-07-10, Sharding EMEA 2023-07-24
    • 5

      Refreshing routing info happens under a lot of circumstances on mongos & mongod, e.g. splitting & moving chunks & shard version Check(when routing requests for read/write queries), etc. Efficiency of refreshing is crucial to MongoDB sharded cluster’s core functionalities.
      In production clusters, chunk number grows rapidly with data keeps flowing in, resulting longer refreshing duration, all client requests are blocked. Although the sql of client requests is simple and the system load (CPU, MEM, IO) is low, client request jitter time has high latency during the route refreshing. For example, a cluster with 1 million chunks, it’d take seconds to do the refresh, severely blocking all client queries.

        1. image-2022-11-26-16-21-42-695.png
          image-2022-11-26-16-21-42-695.png
          58 kB
        2. image-2022-11-26-16-23-01-172.png
          image-2022-11-26-16-23-01-172.png
          58 kB
        3. image-2022-11-26-16-23-18-848.png
          image-2022-11-26-16-23-18-848.png
          184 kB
        4. image-2022-11-26-16-23-53-761.png
          image-2022-11-26-16-23-53-761.png
          173 kB
        5. image-2022-11-26-16-24-26-994.png
          image-2022-11-26-16-24-26-994.png
          111 kB
        6. image-2022-11-26-16-24-40-471.png
          image-2022-11-26-16-24-40-471.png
          111 kB
        7. image-2022-11-26-16-24-57-224.png
          image-2022-11-26-16-24-57-224.png
          133 kB
        8. image-2022-11-26-16-25-41-574.png
          image-2022-11-26-16-25-41-574.png
          149 kB
        9. image-2022-11-26-16-28-47-370.png
          image-2022-11-26-16-28-47-370.png
          49 kB
        10. image-2022-11-26-16-34-25-167.png
          image-2022-11-26-16-34-25-167.png
          342 kB
        11. MongoDB Routing Info Refresh Optimization.pdf
          2.37 MB
        12. MongoDB Routing Info Refresh Optimization-1.pdf
          2.37 MB

            Assignee:
            tommaso.tocci@mongodb.com Tommaso Tocci
            Reporter:
            1147952115@qq.com y yz
            Votes:
            1 Vote for this issue
            Watchers:
            39 Start watching this issue

              Created:
              Updated:
              Resolved: