[SERVER-7515] idempotence violation when intermediate document exceeds size threshold Created: 30/Oct/12  Updated: 22/May/23

Status: Open
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Backlog - Replication Team
Resolution: Unresolved Votes: 2
Labels: bkp, former-robust-initial-sync, idempotency, initialSync
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-33946 Decrease number of initial sync attem... Blocked
Duplicate
is duplicated by SERVER-8505 Document size limit can cause idempot... Closed
is duplicated by SERVER-28709 Initial Sync from MMAP to WT fails if... Closed
is duplicated by SERVER-60160 Initial syncing node can crash due to... Closed
Related
related to SERVER-6399 Refactor update() code Closed
is related to SERVER-4781 replica set initial sync failure when... Closed
Assigned Teams:
Replication
Operating System: ALL
Participants:
Case:
Linked BF Score: 0

 Description   

The update code enforces different size constraints when modifiers are applied, including:

  • The document size cannot exceed the user doc size limit
  • The array backfill amount cannot exceed the array backfill limit
  • Other constraints?

One of these constraints may be violated when an update is applied to a future version of a document present on a secondary, even though the constraint was not violated when the operation was performed on the primary. If this occurs, data from another field may not be properly replicated to the secondary. And if during initial sync of a replica set the initial sync may fail.

Tests:

diff --git a/src/mongo/dbtests/repltests.cpp b/src/mongo/dbtests/repltests.cpp
index 326dffb..0bfafc6 100644
--- a/src/mongo/dbtests/repltests.cpp
+++ b/src/mongo/dbtests/repltests.cpp
@@ -1332,6 +1332,63 @@ public:
     }
 };
 
+
+class DocSizeLimitExceededOnReplay : public Base {
+public:
+    DocSizeLimitExceededOnReplay() : _bigString(10 * 1024 * 1024, 'c') {}
+    void doIt() const {
+        _client.update(ns(), BSONObj(), fromjson("{$set:{z:2}}"));
+        // This update will work when applied initially, but when applied to the new
+        // document with 'b' set, the total doc size will exceed the max user size.
+        _client.update(ns(), BSONObj(), BSON("$set" << BSON("a" << _bigString << "z" << 3)));
+        _client.update(ns(), BSONObj(), fromjson("{$unset:{a:1}}"));
+        _client.update(ns(), BSONObj(), BSON("$set" << BSON("b" << _bigString)));
+    }
+    using ReplTests::Base::check;
+    void check() const {
+        ASSERT_EQUALS(1, count());
+        ASSERT_EQUALS(3, one(BSON("_id" << 0))["z"].number());
+    }
+    void reset() const {
+        deleteAll(ns());
+        insert(BSON("_id" << 0 << "z" << 1));
+    }
+
+private:
+    string _bigString;
+};
+
+
+class ArrayBackfillLimitExceededOnReplay : public Base {
+public:
+    void doIt() const {
+        _client.update(ns(), BSONObj(), fromjson("{$set:{z:1}}"));
+        // Modify an array field with a high index value.  Will work when applied to the
+        // initial doc, but not when applied to the future doc where a is [], due to the
+        // array backfill limit implementation.
+        _client.update(ns(), BSONObj(), fromjson("{$set:{'a.1599999':false,z:2}}"));
+        _client.update(ns(), BSONObj(), fromjson("{$set:{a:[]}}"));
+    }
+
+    using ReplTests::Base::check;
+
+    void check() const {
+        ASSERT_EQUALS(1, count());
+        ASSERT_EQUALS(2, one(BSON("_id" << 0))["z"].number());
+    }
+
+    void reset() const {
+        deleteAll(ns());
+        // Insert a doc with a large array in the 'a' field.
+        BSONArrayBuilder bab;
+        for (int32_t i = 0; i < 1600000; ++i) {
+            bab << true;
+        }
+        insert(BSON("_id" << 0 << "a" << bab.arr()));
+    }
+};
+
+
 }  // namespace Idempotence
 
 class DeleteOpIsIdBased : public Base {
@@ -1448,6 +1505,7 @@ public:
     }
 };
 
+
 class All : public Suite {
 public:
     All() : Suite("repl") {}
@@ -1508,6 +1566,8 @@ public:
         add<Idempotence::AddToSetEmptyMissing>();
         add<Idempotence::ReplaySetPreexistingNoOpPull>();
         add<Idempotence::ReplayArrayFieldNotAppended>();
+        add<Idempotence::DocSizeLimitExceededOnReplay>();
+        add<Idempotence::ArrayBackfillLimitExceededOnReplay>();
         add<DeleteOpIsIdBased>();
         add<DatabaseIgnorerBasic>();
         add<DatabaseIgnorerUpdate>();



 Comments   
Comment by Siyuan Zhou [ 17/Oct/19 ]

Now that we don't fetch missing documents anymore, fixing this will need to change a fundamental size limit of BSON object in the codebase. Since this bug is very rare in the past several years, we are closing it. Feel free to reopen it if this happens again.

Comment by Siyuan Zhou [ 27/Jul/18 ]

Just to add another concrete example. I and U represent insert and update respectively.

I a: {}     // field “a” is an empty sub-document, just to make following ops valid.
--- Start Optime ---
U $set a: { <7MB subdoc> }   // Now the field “a” is a 7 MB sub-document.
U $set b: { <7MB subdoc> }   // The doc has two fields, “a” and “b”, 7 MB each.
U $unset a                   // Only field “b” exists.
U $set c: { <7MB subdoc> }   // The doc has two fields, “b” and “c”, 7 MB each.
--- Clone End Optime ---
When applying the first update again, the size of doc becomes 21 MB, larger than 16 MB, so it’ll fail.

Comment by David Henderson [ 31/Jan/18 ]

I've been hit by this one again, this time doing an initial sync from WT to WT (3.4.10)

Comment by David Henderson [ 26/Apr/17 ]

Yes, I think so - skip the doc after the error, and pull a complete copy at the end.

Comment by Eric Milkie [ 26/Apr/17 ]

I see now. The work for this ticket should be to change the code to treat "document became too large" update errors the same as we current treat "document is missing" update errors.

Comment by David Henderson [ 26/Apr/17 ]

Basically as outlined in the marked-as-dupe SERVER-28709

  • Initial sync fails and retries the whole initial sync from the start - problematic when it fails at 90% after many hours! (With no guarantee that the next attempt will succeed)
Comment by Eric Milkie [ 26/Apr/17 ]

Hi David,
I expected that the temporarily-too-large-document would simply get recopied over and initial sync would continue. Can you explain more about what you are seeing?

Comment by David Henderson [ 26/Apr/17 ]

Any progress on this one? Happened a couple of times now during intial syncs from MMAP -> WT

Comment by Eric Milkie [ 08/Sep/16 ]

To recover from this situation, we need to copy the full document from the sync source. I believe initial sync may already do this, since it treats all update errors with the same behavior; we should confirm this.

Generated at Thu Feb 08 03:14:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.