Details

    • Backwards Compatibility:
      Fully Compatible
    • Backport Completed:
    • Steps To Reproduce:
      Hide

      python buildscripts/resmoke.py --executor no_server repro_server21553.js
      

      Show
      python buildscripts/resmoke.py --executor no_server repro_server21553.js
    • Sprint:
      Build C (11/20/15)

      Description

      • 12 cores, 24 cpus, 64 GB memory
      • 1-member replica set
      • oplog size 5 GB, default cache
      • data on ssd, -journal on separate hdd- no journal
      • 5 threads inserting 1 KB random docs as fast as possible (see code below)
      • recent nightly build:

        2015-11-19T08:41:38.422-0500 I CONTROL  [initandlisten] db version v3.2.0-rc2-211-gbd58ea2
        2015-11-19T08:41:38.422-0500 I CONTROL  [initandlisten] git version: bd58ea2ba5d17b960981990bb97cab133d7e90ed
        

      • periods where oplog size exceeded configured size start at B and D
      • oplog size recovered during stall that started at C, but did not recover after D
      • growth period starting at D appears to coincide with cache-full condition, but excess size starting at B looks like it may be simply related to rate of inserts
      1. benchRun.uncompressible
        2 kB
        Martin Bligh
      2. diagnostic.data.tar
        189 kB
        Bruce Lucas
      3. monitorOplog
        0.2 kB
        Martin Bligh
      4. oplog_overflow.js
        1 kB
        Martin Bligh
      5. repro_server21553.js
        0.8 kB
        Max Hirschhorn
      6. truncate_span.patch
        5 kB
        Max Hirschhorn
      1. 1.png
        502 kB
      2. 2.png
        599 kB
      3. 3.png
        596 kB
      4. 4.png
        567 kB
      5. oplog-grows.png
        156 kB
      6. oplog-unbounded.png
        174 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: SERVER-21553 3.0 backport bug fix.

          Free the referenced block in the right place
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/38369aebcf7da942929a3c27f72495e8dbe3e2d3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: SERVER-21553 3.0 backport bug fix. Free the referenced block in the right place Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/38369aebcf7da942929a3c27f72495e8dbe3e2d3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2354 from wiredtiger/server_21553_30backport

          SERVER-21553 3.0 backport
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/27d0cbdf8046565dba6902f4e6ee93b2642f0d19

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2354 from wiredtiger/server_21553_30backport SERVER-21553 3.0 backport Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/27d0cbdf8046565dba6902f4e6ee93b2642f0d19
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0

          ref: deb2d81..27d0cbd

          6feaa28 WT-2251 Fix leaf of deleted page addresses.
          1da2d3a SERVER-21568 Fix a use-after-free.
          4228295 SERVER-21553 Free blocks during reverse splits.
          0398515 WT-2241 Use a lock to protect transaction ID allocation.
          06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs.
          16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit.
          Branch: v3.0
          https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.0.7-20-g27d0cbd.tar.gz from wiredtiger branch mongodb-3.0 ref: deb2d81..27d0cbd 6feaa28 WT-2251 Fix leaf of deleted page addresses. 1da2d3a SERVER-21568 Fix a use-after-free. 4228295 SERVER-21553 Free blocks during reverse splits. 0398515 WT-2241 Use a lock to protect transaction ID allocation. 06a5c7b WT-2237 Avoid yields if we race allocating transaction IDs. 16a418b WT-2237 Have threads publish unique transaction IDs so that updates always become visible immediately on commit. Branch: v3.0 https://github.com/mongodb/mongo/commit/eb8bc24a0d36b673022faa83976a4b33bc8f676b
          Hide
          michael.cahill Michael Cahill added a comment -

          For the record, there were two underlying issues here in WiredTiger:

          1. When leaf pages were fast-deleted, a subsequent split into the parent of the deleted page was deleting the reference to the page in memory without deleting the block on disk. This caused a leak of space on disk.
          2. In some workloads (including capped collection truncates), internal pages referencing many delete leaves could stay in cache. Changing eviction to visit these internal pages causes the space to be freed on disk.
          Show
          michael.cahill Michael Cahill added a comment - For the record, there were two underlying issues here in WiredTiger: When leaf pages were fast-deleted, a subsequent split into the parent of the deleted page was deleting the reference to the page in memory without deleting the block on disk. This caused a leak of space on disk. In some workloads (including capped collection truncates), internal pages referencing many delete leaves could stay in cache. Changing eviction to visit these internal pages causes the space to be freed on disk.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.2-rc4-70-g197eef0.tar.gz from wiredtiger branch mongodb-3.2

          ref: 7a4f325..197eef0

          bff6525 SERVER-21553 Free blocks during reverse splits.
          c70b097 WT-2253: prioritize WT_READGEN_OLDEST pages for eviction.
          4fc3e39 WT-2253 Evict pages left behind by in-memory splits.
          Branch: master
          https://github.com/mongodb/mongo/commit/f4309efa6d478c1dae79d2483d756c00a069175f

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.2-rc4-70-g197eef0.tar.gz from wiredtiger branch mongodb-3.2 ref: 7a4f325..197eef0 bff6525 SERVER-21553 Free blocks during reverse splits. c70b097 WT-2253 : prioritize WT_READGEN_OLDEST pages for eviction. 4fc3e39 WT-2253 Evict pages left behind by in-memory splits. Branch: master https://github.com/mongodb/mongo/commit/f4309efa6d478c1dae79d2483d756c00a069175f

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile