Change stream busy-spins in poll_next after watched collection is dropped (3.6.0 regression from RUST-2324 cursor rework)

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 3.9.0, 3.8.1
    • Affects Version/s: 3.6.0, 3.7.0, 3.8.0
    • Component/s: None
    • None
    • Rust Drivers
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Polling a ChangeStream after the watched collection has been dropped busy-spins inside the driver on mongodb 3.6.0 through 3.8.0 (latest). After the Drop and Invalidate events are delivered, the next stream.try_next() never completes and never returns Poll::Pending — the poll loop spins hot, pegging one CPU core indefinitely.

      Because the spin never yields, this is worse than a hang for async applications: on a current-thread tokio runtime, sibling timers and tasks are starved outright, so even a tokio::time::timeout wrapped directly around try_next() never fires. In our CI this turned a test that drops a watched collection into a 6-hour job-timeout hang; in production, dropping a watched collection would peg a worker core.

      On 3.5.2 the same program receives Drop, Invalidate, and then the stream ends promptly.

      Steps to reproduce

      Standalone repro (Cargo.lock pinned to 3.8.0, includes docker commands for a single-node replica set): https://github.com/bytenik/mongodb-changestream-drop-spin

      1. Open a change stream on a collection (coll.watch().await)
      2. Drop the collection
      3. Poll the stream in a loop with try_next()

      Observed (3.8.0)

      change stream open; dropping the watched collection…
      [1] event Drop after 3.768625ms
      [2] event Invalidate after 21.375µs
      (spins forever at ~100% CPU; a 10s tokio timeout around try_next never fires)
      

      Expected (behavior of 3.5.2)

      change stream open; dropping the watched collection…
      [1] event Drop after 1.144459ms
      [2] event Invalidate after 10.416µs
      [3] stream ended after 792ns — OK, exiting
      

      Bisection

      3.5.2 OK → 3.6.0 spins (3.7.0 and 3.8.0 too). This points at the 3.6.0 cursor rework (RUST-2324, "Implement Cursor using RawBatchCursor", mongo-rust-driver PR #1620).

      Stack samples

      macOS sample of the spinning process: 1631/1631 samples on-CPU (no parked frames), all inside the change-stream poll path:

      mongodb::change_stream::ChangeStream::poll_next
        mongodb::change_stream::StreamState::poll_next              (change_stream.rs:221)
          mongodb::change_stream::common::CursorWrapper::next_if_any (common.rs:85)
            <Cursor as InnerCursor>::try_advance                    (change_stream.rs:249)
              mongodb::cursor::Cursor::try_advance                  (cursor.rs:202)
                mongodb::cursor::stream::BatchBuffer<Raw>::try_advance      (stream.rs:136-137)
                  mongodb::cursor::stream::BatchBuffer<Raw>::advance_internal
                    mongodb::cursor::raw_batch::RawBatchCursor::poll_next
                    (+ heavy memmove/memcpy churn — actively re-advancing batches,
                     never returning Pending)
      

      It looks like the post-invalidate cursor state keeps try_advance immediately "ready" in a loop instead of surfacing the exhausted/invalidated cursor as stream end (as pre-3.6 versions did) or awaiting a getMore.

      Environment

      Reproduced on macOS (aarch64) and Linux x86_64 CI runners; server mongo:8.0 (8.0.28) single-node replica set; tokio 1.x. Both current-thread and multi-thread runtimes spin; current-thread additionally starves timers.

            Assignee:
            Abraham Egnor
            Reporter:
            David Pfeffer (EXT)
            None
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: