-
Type: Bug
-
Resolution: Duplicate
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Replication, TTL
-
None
-
ALL
-
Execution Team 2020-07-27
-
(copied to CRM)
ISSUE DESCRIPTION AND IMPACT
When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:
2018-12-18T15:19:00.857-0700 F REPL [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\ id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } } 2018-12-18T15:19:00.863-0700 I - [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061
Additional issue: when an update is made to a single document in a capped collection and the document is about to be removed as part of the collection cap, it is possible for a secondary to abort with a similar error.
DIAGNOSIS AND AFFECTED VERSIONS
This can occur when a document is overwritten on the Secondary before it is overwritten on the Primary, and is then subject to two size-neutral updates on two fields.
On the mmapV1 storage engine, differences in storage fragmentation between the Primary and the Secondary are one of the known triggers of this issue.
This can also occur when a document is deleted by capped collection enforcement on the Secondary before an update to that document is replicated from the Primary.
This affects all MongoDB versions 3.2 through 4.4. SERVER-21700 fixes this issue in MongoDB 5.0.
REMEDIATION AND WORKAROUNDS
- If this occurs, resync the affected Secondaries.
- Ensure that capped collections are the same size across the whole replica set.
- Do not update documents that are frequently rolled over capped collections. Consider using a TTL for this use-case instead.
Original description
When multiple (size-neutral) updates are made to a single document in a capped collection, it is possible for a secondary to abort on errors like:
2018-12-18T15:19:00.857-0700 F REPL [repl writer worker 7] writer worker caught exception: 10003 Cannot change the size of a document in a capped collection: 33 != 61 on: { ts: Timestamp 1545171540000|1, t: 3, h: 7402520059475909511, v: 2, op: "u", ns: "test.my_capped", o2: { _\ id: ObjectId('5c12e0f1ea151ba34bd0ae11') }, o: { $set: { y: "a size-neutral value" } } } 2018-12-18T15:19:00.863-0700 I - [repl writer worker 7] Fatal assertion 16359 CannotGrowDocumentInCappedNamespace: Cannot change the size of a document in a capped collection: 33 != 61 at src/mongo/db/repl/sync_tail.cpp 1061
- duplicates
-
SERVER-21700 Do not relax constraints during steady state replication
- Closed
- is duplicated by
-
SERVER-38741 Mongodb replset member fails to start "CannotGrowDocumentInCappedNamespace: Cannot change the size of a document"
- Closed
- is related to
-
SERVER-37172 rollbackViaRefetch should not update documents in capped collections
- Closed
- related to
-
SERVER-8972 extent layout differences can cause incorrect capped collection replication
- Closed