Sharded time-series collMod can return success while leaving metadata inconsistent if the DB-primary participant fails

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Catalog and Routing
    • ALL
    • Hide

      We wrote a JS reproducer named collmod_primary_view_failure_repro.js.

      The reproducer uses normal sharded-cluster operations, plus a one-shot failCommand only to exercise the real participant-command failure path at the relevant point.

      Step 1: Start a two-shard ShardingTest cluster.

      Step 2: Enable sharding on a database whose primary shard is shard0.

      Step 3: Create and shard a time-series collection with granularity seconds.

      Step 4: Move all chunks off the DB-primary shard, so shard0 is the DB primary but owns no chunks for the bucket collection.

      Step 5: Set FCV to the last LTS FCV so the collection uses the legacy time-series view representation.

      Step 6: Verify that the DB-primary legacy view has the old bucket span.

      Step 7: Enable a one-shot failpoint on the DB primary:

      configureFailPoint(

          st.rs0.getPrimary(),

          "failCommand",

          

      {         failCommands: ["_shardsvrCollModParticipant"],         errorCode: ErrorCodes.BadValue,         failInternalCommands: true,     }

      ,

          {times: 1},

      );

      Step 8: Run public collMod through mongos:

      db.runCommand({collMod: "ts", timeseries: {granularity: "hours"}});

      Step 9: Check the config metadata, the DB-primary legacy view pipeline, and checkMetadataConsistency().

      Show
      We wrote a JS reproducer named collmod_primary_view_failure_repro.js. The reproducer uses normal sharded-cluster operations, plus a one-shot failCommand only to exercise the real participant-command failure path at the relevant point. Step 1: Start a two-shard ShardingTest cluster. Step 2: Enable sharding on a database whose primary shard is shard0. Step 3: Create and shard a time-series collection with granularity seconds. Step 4: Move all chunks off the DB-primary shard, so shard0 is the DB primary but owns no chunks for the bucket collection. Step 5: Set FCV to the last LTS FCV so the collection uses the legacy time-series view representation. Step 6: Verify that the DB-primary legacy view has the old bucket span. Step 7: Enable a one-shot failpoint on the DB primary: configureFailPoint(     st.rs0.getPrimary(),     "failCommand",      {         failCommands: ["_shardsvrCollModParticipant"],         errorCode: ErrorCodes.BadValue,         failInternalCommands: true,     } ,     {times: 1}, ); Step 8: Run public collMod through mongos: db.runCommand({collMod: "ts", timeseries: {granularity: "hours"}}); Step 9: Check the config metadata, the DB-primary legacy view pipeline, and checkMetadataConsistency().
    • 🟥 DDL
    • None
    • None
    • None
    • None
    • None
    • None

      Sharded time-series collMod can return success while leaving metadata inconsistent if the DB-primary participant fails

      Description

      For a sharded legacy time-series collection, collMod can return ok: 1 even though the DB-primary shard failed the only participant command that is allowed to update the legacy time-series view definition.

      The observed result is persistent mixed metadata: the config metadata and owning shard bucket metadata advance to the new time-series bucketing options, while the DB-primary legacy view pipeline remains at the old bucket span. checkMetadataConsistency() then reports CollectionOptionsMismatch.

      Analysis

      In the sharded collMod coordinator, the DB primary gets a special _shardsvrCollModParticipant request with performViewChange: true, because the view definition exists only on the primary shard.

      However, if the DB primary owns no chunks, _sendCollModToPrimaryShard() ignores a non-retriable DB-primary participant failure and returns an empty response.

      Later, the coordinator only appends the primary response when the primary owns chunks.

      That seems unsafe for legacy time-series collMod, because the primary participant is still the only participant that can update the view pipeline. The time-series collMod path documents that updating the buckets collection options without updating the view pipeline can cause incorrect query behavior.

      Actual Result

      The public collMod command succeeds even though the DB-primary participant failed:

      [jsTest] collMod result after injected primary participant failure:

      { [jsTest]     "ok" : 1, The DB-primary legacy view remains at the old bucket span: [jsTest] DB-primary legacy view after successful collMod result: [jsTest] \{ "timeField" : "t", "metaField" : "m", "bucketMaxSpanSeconds" : 86400 }

      checkMetadataConsistency() reports mixed metadata:

      [jsTest] observed checkMetadataConsistency output: [

      [jsTest]     "type" : "CollectionOptionsMismatch",

      [jsTest]     "namespace" : "mc1_primary_participant_failure.system.buckets.ts",

      [jsTest]     "granularity" : "minutes",

      [jsTest]     "bucketMaxSpanSeconds" : 86400

      [jsTest]     "db-primary-shard"

      [jsTest]     "granularity" : "hours",

      [jsTest]     "bucketMaxSpanSeconds" : 2592000

      [jsTest]     "config"

      [jsTest]     "owner-shard"

      [jsTest] final mismatch count: 2

      A clean control run, without the injected participant failure, produced no metadata inconsistency:

      [jsTest] after clean collMod metadata inconsistencies: [ ]

      The bad state persisted until the test cleanup dropped the database. I did not observe any later sync, resend, or caller-side guard repairing it before cleanup.

      Expected Result

      For legacy time-series collMod operations that require a primary-only view update, the coordinator should not report success if the DB-primary participant failed to apply that view update.

       

      [1] The bug is found using Specula.

       

        1. test_bugMC-1_primary_participant_failure_ignored.js
          7 kB

            Assignee:
            Unassigned
            Reporter:
            official specula (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: