Mirror Reads loses update filter when update command is rewritten by mongos into inline OP_MSG format

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: 4.4.29, 8.0.23, 5.0.34, 7.0.37, 6.0.30
    • Component/s: None
    • None
    • Networking & Observability
    • ALL
    • Hide

      1. Deploy a sharded cluster consisting of:

         * mongos
         * a shard replica set with primary and secondary nodes

      2. Enable Mirror Reads on the primary node with a sampling rate of 1.0.

      db.adminCommand({
    setParameter: 1,
    mirrorReads: {
        samplingRate: 1.0,
        maxTimeMS: 1000
    }
})

      3. Configure the secondary node to log every mirrored operation by setting slowMS to 0.

      db.setProfilingLevel(0, {slowms: 0});

      4. Create a test collection and an index.

      use mirror_test
db.users.createIndex({ username: 1 })

      5. Insert test data.

      for (let i = 0; i < 100000; i++) {
    db.users.insertOne({
        username: "user_" + i.toString().padStart(6, "0"),
        lastLogin: new Date()
    })
}

      6. Connect to the cluster through mongos using PyMongo.

      7. Execute a filtered update operation.

      db.users.update_one(
    {"username": "user_000021"},
    {"$set": {"lastLogin": datetime.utcnow()}}
)

      8. Examine the secondary node slow query log entries generated by Mirror Reads.

      Expected Result

      The mirrored operation preserves the original update predicate and uses an indexed access path corresponding to:

      { username: "user_000021" }

      Actual Result

      The mirrored operation recorded in the secondary node slow query log does not contain the original update predicate and is executed as a COLLSCAN.

      Additional Observation

      The issue is reproducible when the update operation is routed through mongos.

      Executing the same update directly against mongod does not reproduce the issue; the mirrored operation retains the original predicate and does not perform a collection scan.

      Show
      1. Deploy a sharded cluster consisting of:    * mongos    * a shard replica set with primary and secondary nodes 2. Enable Mirror Reads on the primary node with a sampling rate of 1.0. db.adminCommand({     setParameter: 1,     mirrorReads: {         samplingRate: 1.0,         maxTimeMS: 1000     } }) 3. Configure the secondary node to log every mirrored operation by setting slowMS to 0. db.setProfilingLevel(0, {slowms: 0}); 4. Create a test collection and an index. use mirror_test db.users.createIndex({ username: 1 }) 5. Insert test data. for (let i = 0; i < 100000; i++) {     db.users.insertOne({         username: "user_" + i.toString().padStart(6, "0" ),         lastLogin: new Date()     }) } 6. Connect to the cluster through mongos using PyMongo. 7. Execute a filtered update operation. db.users.update_one(     { "username" : "user_000021" },     { "$set" : { "lastLogin" : datetime.utcnow()}} ) 8. Examine the secondary node slow query log entries generated by Mirror Reads. Expected Result The mirrored operation preserves the original update predicate and uses an indexed access path corresponding to: { username: "user_000021" } Actual Result The mirrored operation recorded in the secondary node slow query log does not contain the original update predicate and is executed as a COLLSCAN. Additional Observation The issue is reproducible when the update operation is routed through mongos. Executing the same update directly against mongod does not reproduce the issue; the mirrored operation retains the original predicate and does not perform a collection scan.
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When Mirror Reads is enabled in a sharded cluster, update operations routed through mongos may lose their update predicate during mirror request generation.

      We observed different behavior between:

      • Client -> mongod
      • Client -> mongos -> mongod

      For the same update operation, direct connections to mongod correctly preserve the update filter in mirrored requests. However, when the request is routed through mongos, the mirrored request generated on the secondary appears to lose the original filter predicate, resulting in a collection scan.

      Packet captures show that the root cause is related to different but semantically equivalent OP_MSG encodings.

      PyMongo sends update commands using an OP_MSG Document Sequence:

       

      {{Body
      Document Sequence ("updates")}}

      In this form, Mirror Reads correctly extracts the update predicate from the updates section.

      After the request passes through mongos, mongos rewrites the command into an inline OP_MSG representation:

       

      {{{
      update: "users",
      updates: [
      {
      q:

      { username: "user_0094" }

      ,
      u: { $set:

      { lastLogin: ... }

      }
      }
      ]
      }}}

      In this representation, the updates information is embedded directly in the Body section and no Document Sequence exists.

      Although both encodings are valid OP_MSG representations of the same update command, Mirror Reads appears to only extract predicates from the Document Sequence representation. When the command is rewritten into inline format by mongos, the predicate is no longer correctly propagated into the mirrored request.

      As a result, filtered update operations routed through mongos may generate mirrored reads without the original filter, causing unnecessary collection scans on secondary nodes and creating behavior that differs from direct-to-mongod execution.

      Mirror Reads should generate equivalent mirrored requests regardless of whether the update command is encoded using:

      • OP_MSG Document Sequence ("updates")
      • Inline updates array in the Body section

      Both representations are protocol-valid and semantically equivalent.

            Assignee:
            Unassigned
            Reporter:
            Li GuangZhi (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: