Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38724

Under rare conditions, updates to capped collections can cause SECONDARY members to fail

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Replication, TTL
    • None
    • ALL
    • Execution Team 2020-07-27

      Issue Status as of April 13, 2021

      ISSUE DESCRIPTION AND IMPACT
      When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:

      2018-12-18T15:19:00.857-0700 F REPL     [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\
      id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } }
      2018-12-18T15:19:00.863-0700 I -        [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061 

      Additional issue: when an update is made to a single document in a capped collection and the document is about to be removed as part of the collection cap, it is possible for a secondary to abort with a similar error.
      DIAGNOSIS AND AFFECTED VERSIONS
      This can occur when a document is overwritten on the Secondary before it is overwritten on the Primary, and is then subject to two size-neutral updates on two fields.
      On the mmapV1 storage engine, differences in storage fragmentation between the Primary and the Secondary are one of the known triggers of this issue.

      This can also occur when a document is deleted by capped collection enforcement on the Secondary before an update to that document is replicated from the Primary.

      This affects all MongoDB versions 3.2 through 4.4. SERVER-21700 fixes this issue in MongoDB 5.0.

      REMEDIATION AND WORKAROUNDS

      • If this occurs, resync the affected Secondaries.
      • Ensure that capped collections are the same size across the whole replica set.
      • Do not update documents that are frequently rolled over capped collections. Consider using a TTL for this use-case instead.
      Original description

      When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:

      2018-12-18T15:19:00.857-0700 F REPL     [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\
      id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } }
      2018-12-18T15:19:00.863-0700 I -        [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061 

            Assignee:
            gregory.noma@mongodb.com Gregory Noma
            Reporter:
            mark.brinsmead@mongodb.com Mark Brinsmead
            Votes:
            3 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: