Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38724

Under rare conditions, updates to capped collections can cause SECONDARY members to fail

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication, TTL
    • Labels:
      None
    • Operating System:
      ALL
    • Sprint:
      Execution Team 2020-07-27
    • Case:

      Description

      Issue Status as of April 13, 2021

      ISSUE DESCRIPTION AND IMPACT
      When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:

      2018-12-18T15:19:00.857-0700 F REPL     [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\
      id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } }
      2018-12-18T15:19:00.863-0700 I -        [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061 

      Additional issue: when an update is made to a single document in a capped collection and the document is about to be removed as part of the collection cap, it is possible for a secondary to abort with a similar error.
      DIAGNOSIS AND AFFECTED VERSIONS
      This can occur when a document is overwritten on the Secondary before it is overwritten on the Primary, and is then subject to two size-neutral updates on two fields.
      On the mmapV1 storage engine, differences in storage fragmentation between the Primary and the Secondary are one of the known triggers of this issue.

      This can also occur when a document is deleted by capped collection enforcement on the Secondary before an update to that document is replicated from the Primary.

      This affects all MongoDB versions 3.2 through 4.4. SERVER-21700 fixes this issue in MongoDB 5.0.

      REMEDIATION AND WORKAROUNDS

      • If this occurs, resync the affected Secondaries.
      • Ensure that capped collections are the same size across the whole replica set.
      • Do not update documents that are frequently rolled over capped collections. Consider using a TTL for this use-case instead.
      Original description

      When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:

      2018-12-18T15:19:00.857-0700 F REPL     [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\
      id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } }
      2018-12-18T15:19:00.863-0700 I -        [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061 

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              gregory.noma Gregory Noma
              Reporter:
              mark.brinsmead Mark Brinsmead
              Participants:
              Votes:
              3 Vote for this issue
              Watchers:
              29 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: