Context

During search executor shutdown, shutdownTaskExecutor() drains the registered PinnedConnectionTaskExecutors (PCTEs) before shutting down the underlying executor:

// src/mongo/db/query/search/search_task_executors.cpp
shutdownPinnedExecutors(svc, execPtr);
...
execPtr->shutdown();
execPtr->join();

shutdownPinnedExecutors() (src/mongo/executor/pinned_connection_task_executor_registry.cpp) scans the registry under the registry lock and skips entries whose pinned weak_ptr is expired:

// The token erases its entry asynchronously, so the pinned weak_ptr can already be
// expired here even though the underlying executor is still alive. Skip those.
>;if (auto pinned = entry.pinned.lock()) {
    toShutdown.emplace_back(std::move(pinned));
}

Problem (confirmed)

An expired weak_ptr only means the strong ref count reached zero; it does not mean the PCTE has finished shutting down. ~PinnedConnectionTaskExecutor() runs shutdown() then join(), and join() waits for the request queue to drain before joining the underlying executor — so the destructor can still be in flight (waiting for an in-flight remote command's cancellation to complete) at the moment the scan skips its entry.

Because shutdownTaskExecutor() shuts down the underlying executor right after the drain, the pending cancellation continuation is then scheduled on a shut-down executor. The continuation runs on makeGuaranteedExecutor(baton, _cancellationExecutor) where _cancellationExecutor is the underlying executor; TaskExecutor::schedule runs a task inline with the failure status when the executor rejects work, so GuaranteedExecutor::enforceRunOnce invariants with OutOfLineExecutor rejected work.

Reproduction

Confirmed deterministically with a unit test in src/mongo/executor/pinned_connection_task_executor_test.cpp (10/10 runs abort):

1. A PCTE has an in-flight RPC (message sunk, response pending), so _isDoingNetworking is true.
2. Its last strong reference is dropped: the destructor calls shutdown() (cancelling the RPC via client->cancel()) and then blocks in join().
3. At that instant the registry entry's pinned weak_ptr is expired but teardown is not complete. Draining with shutdownPinnedExecutors() skips the entry; the underlying executor is then shut down (as shutdownTaskExecutor() does).
4. The cancelled RPC's completion is scheduled on the shut-down underlying executor.

Resulting invariant:

Invariant failure  expr="status"  msg="OutOfLineExecutor rejected work"
error="ShutdownInProgress: TaskExecutor shutdown in progress"
location=src/mongo/util/out_of_line_executor.h:123
         (GuaranteedExecutor::enforceRunOnce)

This is the same crash class as AF-21589 / AF-4442, which SERVER-135154 set out to close. The window is narrow (it opens when the underlying executor is busy or shutting down while a cursor teardown races shutdown), which is why it was not caught earlier.

Fix direction

• Track teardown completion for expired entries (e.g. a destruction-complete state on the registry entry or token) and await it outside the registry lock before shutdownTaskExecutor() proceeds to the underlying executor shutdown; or keep the entry drainable until destruction completes.
• Add a regression test where the PCTE destructor is still in progress during the scan — the test added by SERVER-135154 (ShutdownPinnedExecutorsSkipsExpiredPinnedExecutor) only covers destruction that has already completed.

Notes

• Not covered by SERVER-135000 (gRPC transport-layer lifetime; fails later, at transport shutdown) or SERVER-135852 (idle-cursor disposal detection gap).