ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • ALL
    • CAR Team 2026-09-28, CAR Team 2026-10-12, CAR Team 2026-10-26, CAR Team 2026-11-09
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      After investigating several spikes of recovery times from the dashboard as part of HELP-100264, it was concluded that all of the spikes that exceed 1sec follow the same pattern.

      There was a recovery initiated, but it was interrupted by the user/automation in Atlas, which was observed through the following log in the mongod:

      {"t":\{"$date":"..."},"s":"I",  "c":"COMMAND",  "id":..., "svc":"S", "ctx":"...","msg":"Terminating via shutdown command","attr":\{"force":true,"timeoutMillis":15000}}
      

      For the next 15 seconds, the cluster wasn't accepting any requests and doing any work, reporting that it entered the quiesce mode, reporting `remainingQuiesceTimeMillis` as the remaining time in the state, before termination.

      {"t":{"$date":"..."},"s":"I",  "c":"-",        "id":..., "svc":"-", "ctx":"NetworkInterfaceTL-ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"...","error":"ShutdownInProgress: The server is in quiesce mode and will shut down","replicaSet":"...,"response":{"..."},"ok":0,"errmsg":"The server is in quiesce mode and will shut down","code":91,"codeName":"ShutdownInProgress","remainingQuiesceTimeMillis":14710,"$clusterTime":{"clusterTime":{"$timestamp":{"t":...,"i":...}}},"$configTime":{"$timestamp":{"t":...,"i":...}},"$topologyTime":{"$timestamp":{"t":...,"i":...}},"operationTime":{"$timestamp":{"t":...,"i":...}}}}}
      

      After that it proceeds with the shut down and the host becomes unreachable, at this point in time there usually are another series of attempts to recover the disk, however for another few minutes after the server starts, it refuses all the requests and the metadata recovery attempts result in time outs:

      {"t":{"$date":"..."},"s":"I",  "c":"SHARDING", "id":...,   "svc":"S", "ctx":"...","msg":"Failed to refresh metadata for collection","attr":{"namespace":"...","error":"MaxTimeMSExpired: operation exceeded time limit"}}
      

      A bit later (usually it takes another couple of minutes before the server starts accepting executing requests) it finally arrives at a stable timestamp:

      {"t":{"$date":"..."},"s":"I",  "c":"SHARDING", "id":...,"svc":"S", "ctx":"MetadataSynchronizer","msg":"Wait for stable timestamp finished, proceeding to read disk contents","attr":{"collection":"..."}}
      

      and shortly succeeds in recovering the collection:

      {"t":{"$date":"..."},"s":"I",  "c":"SHARDING", "id":...,"svc":"S", "ctx":"MetadataSynchronizer","msg":"Disk contents have been read","attr":{"collection":"..."}},
      ...
      {"t":{"$date":"..."},"s":"I",  "c":"SHARDING", "id":...,"svc":"S", "ctx":"...","msg":"Authoritative collection metadata recovery completed successfully","attr":{"namespace":"...","outcome":"versionMatched","durationMillis":2}}
      

      Therefore there are several moving parts which may affect the calculation of the totalDiskRecoveryMillis statistics reported in the dashboard:

      • adding the time of the server termination and the restart to the statistics
      • adding the time of waiting for the stable timestamp to the statistics

      Additionally, when the recoveries at the same time, the dashboard currently collides them into a single statistic, which is misleading.

      In order to address all of the issues listed and not confuse observers of the dashboard we will fix the reporting mechanism of the totalDiskRecoveryMillis, and as a interim solution will change the grafana dashboard to exclude from the statistics the first refresh that appears after the node restart.

            Assignee:
            Unassigned
            Reporter:
            Rustam Gusein-zade
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: