mapReduce fails with "collection dropped" QueryPlanKilled when balancer resharding races foreground query

XMLWordPrintableJSON

    • Query Integration
    • Fully Compatible
    • ALL
    • v9.0
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      jstests/core/query/js/js_object_properties.js intermittently fails under the sharding_jscore_passthrough_with_balancer and sharding_jscore_passthrough_with_balancer_rate_limited resmoke suites with:

      Error: map reduce failed:{
        "ok" : 0,
        "errmsg" : "MapReduce internal error :: caused by :: Executor error during aggregate command on namespace: js_object_properties.test :: caused by :: collection dropped. UUID <uuid>",
        "code" : 175,
        "codeName" : "QueryPlanKilled",
        ...
      }
      

      Seen twice now: BF-45031 (open) and BF-44125 (closed a month prior with the identical signature — that closure appears to have been premature; its "Fix Revision" link is to SERVER-127318, a MozJS WASM bridge/memory ticket, unrelated to this failure, and its "BF Root Cause Commit" field was never actually set).

      Root cause

      Confirmed from the mongod log of the BF-45031 failing task. These passthrough suites implicitly shard core-suite collections and run a live balancer. The test's collection is created as a single-chunk "unsplittable"/untracked collection. Within ~170ms of creation, the balancer decides to redistribute it via moveCollection, which is implemented internally as a full reshardCollection:

      06:10:38.937  createCollection js_object_properties.test  uuid=c5292091-...
      06:10:39.253  _configsvrReshardCollection "js_object_properties.test"  provenance:"balancerMoveCollection" forceRedistribution:true
      06:10:39.253  _shardsvrReshardCollection ...  provenance:"balancerMoveCollection"
      

      Resharding clones the collection into a temp namespace (js_object_properties.system.resharding.<uuid>) and atomically swaps it in for the original, which changes the collection's UUID — by design, this invalidates any in-flight cursor pinned to the old collection instance. The test's second mapReduce call (translated internally into a multi-stage aggregate) loses its cursor to that swap and surfaces the "collection dropped" QueryPlanKilled error.

      This is a legitimate race between intended balancer behavior and a foreground command that isn't designed to retry around it — not query-execution memory corruption or an actual accidental drop.

      Why this isn't already handled

      These suites already load jstests/libs/override_methods/implicitly_retry_on_migration_in_progress.js specifically so foreground commands transparently retry when they lose a race with sharding background operations. That override does not currently retry mapReduce on this particular "collection dropped via resharding swap" signature.

      Proposed fix

      Extend the existing retry-on-transient-error override (or add a narrowly-scoped new override following the same pattern) so that mapReduce retries when it hits QueryPlanKilled with a "collection dropped" errmsg caused by a concurrent resharding swap, mirroring how migration-in-progress races are already tolerated in this suite. A draft patch is attached/forthcoming; if on closer inspection the retry can't be done safely or cleanly (e.g. mapReduce doesn't route through an interceptable path, or the signature can't be safely distinguished from a real accidental drop), the fallback is to tag the affected test(s) incompatible with the balancer/moveCollection passthroughs instead, similar to the existing incompatibility tags already present in js_object_properties.js.

      Related

      • BF-45031 — current open occurrence
      • BF-44125 — prior occurrence, closed without addressing root cause

            Assignee:
            Charlie Swanson
            Reporter:
            Charlie Swanson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: