blockTimeMS does not reliably block for the full duration

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Server Programmability
    • ALL
    • Hide
      1. Run a mongo instance (single, replica, or sharded – does not seem to matter) on RHEL 8.
      2. Configure a blockTimeMS failpoint to block for X milliseconds.
      3. Let start be the current time from a monotonic clock.
      4. Run a command.
      5. Let end be the current time from a monotonic clock.
      6. Check that end - start is greater than or equal to the requested block time.
      7. Repeat for several iterations until the above check fails.
      Show
      Run a mongo instance (single, replica, or sharded – does not seem to matter) on RHEL 8. Configure a blockTimeMS failpoint to block for X milliseconds. Let start be the current time from a monotonic clock. Run a command. Let end be the current time from a monotonic clock. Check that end - start is greater than or equal to the requested block time. Repeat for several iterations until the above check fails.
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The blockTimeMS failpoint does not seem to reliably block for the full duration. I suspect this is due to the fact that blockTimeMS is implemented using a non-monotonic clock.

      Take the following C++ driver example:

      static constexpr auto block_time = std::chrono::milliseconds(50);
      
      db.run_command(make_document(
          kvp("configureFailPoint", "failCommand"),
          kvp("mode", make_document(kvp("times", 1))),
          kvp("data",
              make_document(
                  kvp("failCommands", bsoncxx::builder::basic::make_array("ping")),
                  kvp("blockTimeMS", std::chrono::milliseconds(block_time).count()),
                  kvp("blockConnection", true)))));
      
      auto const start = std::chrono::steady_clock::now();
      
      db.run_command(make_document(kvp("ping", 1)));
      
      auto const end = std::chrono::steady_clock::now();
      
      auto const elapsed = end - start;
      
      REQUIRE(elapsed >= block_time); // Fails!

      On RHEL 8, the elapsed duration is consistently just shy (<1ms) of the block time.

      Using the full example here, I was able to reproduce this on Evergreen with the following output:

      [2026/02/11 10:21:57.682] ../src/mongocxx/test/v_noabi/block_time_ms_bug_repro.cpp:52: failed: to_raw_us(elapsed) >= to_raw_us(block_time) for: 49365.36299999999755528 >= 50000.0 with 1 message: 'iteration := 0'
      [2026/02/11 10:21:57.732] ../src/mongocxx/test/v_noabi/block_time_ms_bug_repro.cpp:52: failed: to_raw_us(elapsed) >= to_raw_us(block_time) for: 49272.5 >= 50000.0 with 1 message: 'iteration := 1'
      [2026/02/11 10:21:57.782] ../src/mongocxx/test/v_noabi/block_time_ms_bug_repro.cpp:52: failed: to_raw_us(elapsed) >= to_raw_us(block_time) for: 49332.54299999999784632 >= 50000.0 with 1 message: 'iteration := 2'
      [2026/02/11 10:21:57.832] ../src/mongocxx/test/v_noabi/block_time_ms_bug_repro.cpp:52: failed: to_raw_us(elapsed) >= to_raw_us(block_time) for: 49312.42100000000209548 >= 50000.0 with 1 message: 'iteration := 3'

      This pattern continues the majority of the 100 iterations. To be clear, the above duration values are in microseconds. Since the client-side code is measuring the duration using std::chrono::steady_clock, I feel confident that these duration measurements are accurate.

      I suspected this issue was caused by the server basing the sleep duration off the system clock, which is subject to time jumps. After investigating, this appears to be the case:

      From the gettimeofday documentation:

      The time returned by gettimeofday() is affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the system time). If you need a monotonically increasing clock, see clock_gettime(2).

      To correctly implement blockTimeMS, the server should be measuring the duration using a steady, monotonic clock. Otherwise, there are technically no guarantees that the block duration will be honored.

      In practice, blockTimeMS seems to get close enough to the correct block duration most of the time, which is perhaps why this issue has gone undetected. Additionally, I could only ever reproduce this issue on RHEL 8 – I am not sure why.

            Assignee:
            Unassigned
            Reporter:
            Connor MacDonald
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: