Idle $search cursors on views are not disposed at shutdown and stall mongod for the executor timeout

ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: None
    • DB Integration & Observability
    • ALL
    • Hide

      python buildscripts/resmoke.py run --suites=search \
      --installDir=bazel-bin/install-dist-test/bin --multiversionDir=bazel-bin/install-dist-test/bin \
      jstests/with_mongot/search_mocked/repro_view_zero_initial_batch.js

      /**

      • Repro: a $search cursor opened with batchSize: 0 against a VIEW is not disposed at shutdown,
      • so the mongot executor wait hits searchTaskExecutorShutdownTimeoutMS (instead of shutting down
      • promptly like the direct-collection case).
        *
      • @tags: [ requires_fcv_90 ]
        */
        import {describe, it}

        from "jstests/libs/mochalite.js";
        import

        {MongotMock}

        from "jstests/with_mongot/mongotmock/lib/mongotmock.js";

      describe("repro view zero initial batch", function () {
      it("view $search cursor with batchSize: 0", function () {
      const dbName = jsTestName();
      const collName = jsTestName();
      const viewName = collName + "_view";

      const mongotmock = new MongotMock();
      mongotmock.start();

      const conn = MongoRunner.runMongod({
      setParameter:

      { mongotHost: mongotmock.getConnection().host, searchTaskExecutorShutdownTimeoutMS: 3000, }

      ,
      });
      const db = conn.getDB(dbName);
      const coll = db[collName];
      coll.insertMany([{_id: 1, title: "cakes"}, {_id: 2, title: "cookies and cakes"}]);

      const searchQuery =

      {query: "cakes", path: "title"}

      ;
      assert.commandWorked(db.createView(viewName, collName, [{$search: searchQuery}]));

      mongotmock.setMockResponses(
      [
      {
      expectedCommand: {
      search: collName,
      collectionUUID: db.getCollectionInfos(

      {name: collName}

      )[0].info.uuid,
      query: searchQuery,
      $db: dbName,
      },
      response: {
      ok: 1,
      cursor:

      {id: NumberLong(0), ns: coll.getFullName(), nextBatch: []}

      ,
      },
      },
      ],
      NumberLong(1),
      );

      // Aggregate the VIEW with batchSize: 0 -> empty first batch, live cursor, never executes.
      const res = assert.commandWorked(
      db.runCommand({aggregate: viewName, pipeline: [], cursor: {batchSize: 0}}),
      );
      assert.eq(res.cursor.firstBatch.length, 0, "expected an empty first batch");
      assert.neq(res.cursor.id, NumberLong(0), "expected a live cursor");

      const start = Date.now();
      assert.eq(MongoRunner.stopMongod(conn), MongoRunner.EXIT_CLEAN, "unclean shutdown");
      const elapsedMs = Date.now() - start;
      jsTest.log.info("mongod shutdown complete",

      {elapsedMs}

      );
      // BUG: view cursor not disposed -> waits out the 3s backstop.
      assert.lt(elapsedMs, 2500, "view cursor NOT disposed; shutdown hit the executor timeout");

      mongotmock.stop();
      });
      });

      Show
      python buildscripts/resmoke.py run --suites=search \ --installDir=bazel-bin/install-dist-test/bin --multiversionDir=bazel-bin/install-dist-test/bin \ jstests/with_mongot/search_mocked/repro_view_zero_initial_batch.js /** Repro: a $search cursor opened with batchSize: 0 against a VIEW is not disposed at shutdown, so the mongot executor wait hits searchTaskExecutorShutdownTimeoutMS (instead of shutting down promptly like the direct-collection case). * @tags: [ requires_fcv_90 ] */ import {describe, it} from "jstests/libs/mochalite.js"; import {MongotMock} from "jstests/with_mongot/mongotmock/lib/mongotmock.js"; describe("repro view zero initial batch", function () { it("view $search cursor with batchSize: 0", function () { const dbName = jsTestName(); const collName = jsTestName(); const viewName = collName + "_view"; const mongotmock = new MongotMock(); mongotmock.start(); const conn = MongoRunner.runMongod({ setParameter: { mongotHost: mongotmock.getConnection().host, searchTaskExecutorShutdownTimeoutMS: 3000, } , }); const db = conn.getDB(dbName); const coll = db [collName] ; coll.insertMany( [{_id: 1, title: "cakes"}, {_id: 2, title: "cookies and cakes"}] ); const searchQuery = {query: "cakes", path: "title"} ; assert.commandWorked(db.createView(viewName, collName, [{$search: searchQuery}] )); mongotmock.setMockResponses( [ { expectedCommand: { search: collName, collectionUUID: db.getCollectionInfos( {name: collName} ) [0] .info.uuid, query: searchQuery, $db: dbName, }, response: { ok: 1, cursor: {id: NumberLong(0), ns: coll.getFullName(), nextBatch: []} , }, }, ], NumberLong(1), ); // Aggregate the VIEW with batchSize: 0 -> empty first batch, live cursor, never executes. const res = assert.commandWorked( db.runCommand({aggregate: viewName, pipeline: [], cursor: {batchSize: 0}}), ); assert.eq(res.cursor.firstBatch.length, 0, "expected an empty first batch"); assert.neq(res.cursor.id, NumberLong(0), "expected a live cursor"); const start = Date.now(); assert.eq(MongoRunner.stopMongod(conn), MongoRunner.EXIT_CLEAN, "unclean shutdown"); const elapsedMs = Date.now() - start; jsTest.log.info("mongod shutdown complete", {elapsedMs} ); // BUG: view cursor not disposed -> waits out the 3s backstop. assert.lt(elapsedMs, 2500, "view cursor NOT disposed; shutdown hit the executor timeout"); mongotmock.stop(); }); });
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      An idle $search cursor opened with batchSize: 0 against a view (view definition contains $search) is not marked as holding the mongot task executor, so it is not disposed during shutdown. mongod then stalls for the full searchTaskExecutorShutdownTimeoutMS backstop (default 60s), or indefinitely if the timeout is disabled (0).

      Background

      SERVER-129849 added ClientCursorParams::mayHoldMongotTaskExecutor and disposes idle cursors holding mongot executor references before shutting down the search task executor. The flag is set in registerCursor() from aggExState.getOriginalLiteParsedPipeline().hasMongotStage() OR CurOp::debug().mongotCursorId.

      Steps to reproduce

      1. Start mongod with a mongotmock host and searchTaskExecutorShutdownTimeoutMS=3000.
      2. Create a collection and a view whose definition is [{$search: {...}}].
      3. Run db.runCommand({aggregate: <view>, pipeline: [], cursor: {batchSize: 0)}}; it returns an empty first batch and a live cursor.
      4. MongoRunner.stopMongod(conn) without issuing a getMore.

      Expected

      Shutdown disposes the idle view cursor and completes promptly (< ~1s), as the direct-collection case does.

      Actual

      Shutdown waits out the executor timeout:

      • disposeIdleMongotCursorsMillis: 0
      • shutDownSearchTaskExecutorsMillis: 3002
      • Log 12984901 "Timed out waiting for the search task executor to be destroyed during shutdown" at 3000ms
      • Total shutdown ~3.8s

      Control (direct collection, same pipeline, batchSize: 0) passes: shutdown ~710ms, disposeIdleMongotCursorsMillis: 1, no timeout.

      Root cause

      runAggregate resolves the main view and reassigns aggExState to a ResolvedViewAggExState (src/mongo/db/commands/query_cmd/run_aggregate.cpp:1643-1664). ResolvedViewAggExState overrides getOriginalLiteParsedPipeline() to return the caller's unexpanded pipeline (aggregation_execution_state.h:399) — the empty pipeline the user sent — not the resolved pipeline containing $search. So hasMongotStage() is false at run_aggregate.cpp:254. With batchSize: 0 the pipeline never executes, so OpDebug::mongotCursorId is also unset. Both detection paths miss, mayHoldMongotTaskExecutor stays false, and CursorManager::disposeIdleMongotCursorsForShutdown skips the cursor.

      Impact

      Any aggregation against a view that opens an idle search cursor can stall shutdown by the executor timeout (default 60s; hang if disabled). Reproduced on master (d008e6228f5) and on the v9.0/v8.3 backports.

      Fix direction

      Make the ownership check use the resolved/execution pipeline instead of getOriginalLiteParsedPipeline(). Add a regression test (view + $search + batchSize: 0 asserting prompt shutdown).

            Assignee:
            Unassigned
            Reporter:
            Mariano Shaar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: