Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6984

Initial sync can fail, or break future replication, when updates shrink or grow docs in capped collections

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: Storage
    • Labels:
    • Backwards Compatibility:
      Minor Change
    • Operating System:
      ALL

      Description

      Right now we allow updates on docs in capped collections, as long as docs don't grow past the size of their initial allocation. However, during initial sync, the data is cloned but the initial allocation size is lost on the secondary. So if inserts or updates which affect document size occur during the cloning process, when replaying the docs you can get an error message saying "objects in a capped ns cannot grow."

      For instance, this happens if the following sequence of ops happens during initial sync:

      • db.foo.insert( { a : 1, b : "big "}

        )

      • db.foo.update( { a : 1 }

        , {$unset : {b : 1}})

      If the smaller version of the doc is cloned during the initial sync, you will get an error message at the end of the initial sync when it goes to apply the ops:

      Sun Sep  9 19:52:48 [repl writer worker 1] ERROR: exception: failing update: objects in a capped ns cannot grow on: { ts: Timestamp 1347234740000|1, h: -4246821095103890152, op: "i", ns: "test.zzz", o: { _id: ObjectId('504d2bb46bbaa186f1cc7566'), a: 1.0, b: "big" } }
      Sun Sep  9 19:52:48 [repl writer worker 1]   Fatal Assertion 16361
      0x109c45b1b 0x10a09bde7 0x10a072239 0x109df1bc8 0x109e2c915 0x7fff8c2c5782 0x7fff8c2b21c1 
       0   mongod                              0x0000000109c45b1b _ZN5mongo15printStackTraceERSo + 43
       1   mongod                              0x000000010a09bde7 _ZN5mongo13fassertFailedEi + 151
       2   mongod                              0x000000010a072239 _ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE + 393
       3   mongod                              0x0000000109df1bc8 _ZN5mongo10threadpool6Worker4loopEv + 138
       4   mongod                              0x0000000109e2c915 thread_proxy + 229
       5   libsystem_c.dylib                   0x00007fff8c2c5782 _pthread_start + 327
       6   libsystem_c.dylib                   0x00007fff8c2b21c1 thread_start + 13
      Sun Sep  9 19:52:48 [repl writer worker 1] 
       
      ***aborting after fassert() failure

      Even if initial sync manages to succeed, it's possible that future updates which grow a document on the primary will break replication, because there is no space for the doc to grow on the secondary. I've verified that this can occur, and the error message is similar to above.

        Issue Links

          Activity

          Hide
          matulef Kevin Matulef (Inactive) added a comment -

          Options for fixing this are:

          1. when cloning capped collections, allocate the same amount of space the doc was originally allocated with (don't compact).
          2. for capped collections, disallow updates which shrink or grow docs (strictly in-place updates only).
          Show
          matulef Kevin Matulef (Inactive) added a comment - Options for fixing this are: when cloning capped collections, allocate the same amount of space the doc was originally allocated with (don't compact). for capped collections, disallow updates which shrink or grow docs (strictly in-place updates only).
          Hide
          aaron Aaron Staple (Inactive) added a comment -

          Also see SERVER-4939 (with test).

          Show
          aaron Aaron Staple (Inactive) added a comment - Also see SERVER-4939 (with test).
          Hide
          aaron Aaron Staple (Inactive) added a comment -

          Kevin Matulef Sure I'll make the other a dup since you have more info here.

          Show
          aaron Aaron Staple (Inactive) added a comment - Kevin Matulef Sure I'll make the other a dup since you have more info here.
          Hide
          aaron Aaron Staple (Inactive) added a comment -

          Here's a (potentially old) test (from a duplicate ticket):

          Author:

          {u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

          Message: SERVER-4939 test
          Branch: master
          https://github.com/mongodb/mongo/commit/921c3d74f30927ff49261cf79266fbbc9f37901a

          Show
          aaron Aaron Staple (Inactive) added a comment - Here's a (potentially old) test (from a duplicate ticket): Author: {u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'} Message: SERVER-4939 test Branch: master https://github.com/mongodb/mongo/commit/921c3d74f30927ff49261cf79266fbbc9f37901a
          Hide
          burley David Burley added a comment -

          Any update on this issue? Are there any workarounds for it in the mean time?

          Show
          burley David Burley added a comment - Any update on this issue? Are there any workarounds for it in the mean time?
          Hide
          rosmo Taneli Leppä added a comment -

          This affects 2.4.12 as well. Any decent workaround?

          Show
          rosmo Taneli Leppä added a comment - This affects 2.4.12 as well. Any decent workaround?
          Hide
          asya Asya Kamsky added a comment -

          Taneli Leppä - capped collections are most effective when they are used as append-only inserts and tailing reads - they are not as effective for updates, so certainly eliminating update operations would avoid this issue. If updates cannot be avoided then making sure that updates don't change the size of the document would be the next best thing.

          Asya

          Show
          asya Asya Kamsky added a comment - Taneli Leppä - capped collections are most effective when they are used as append-only inserts and tailing reads - they are not as effective for updates, so certainly eliminating update operations would avoid this issue. If updates cannot be avoided then making sure that updates don't change the size of the document would be the next best thing. Asya
          Hide
          milkie Eric Milkie added a comment -

          Due to SERVER-20529, fixed in 3.2.0-rc0, documents in capped collections can no longer shrink, so the initial sync problem is no longer an issue.

          Show
          milkie Eric Milkie added a comment - Due to SERVER-20529 , fixed in 3.2.0-rc0, documents in capped collections can no longer shrink, so the initial sync problem is no longer an issue.

            People

            • Votes:
              11 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: