Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18875

Oplog performance on WT degrades over time after accumulation of deleted items

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.5, 3.1.6
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      Quint Iteration 5, Quint Iteration 6

      Description

      Issue Status as of Jul 14, 2015

      ISSUE SUMMARY AND IMPACT
      Capped collection handling in WiredTiger is inefficient because of way that WiredTiger tracks and expires documents in the capped collection.

      WiredTiger uses a specific internal cursor to find the "beginning of the capped collection". Combined with asynchronous deletion of expired capped collection records, this is inefficient for collections with high numbers of inserts because requests have to process large number of expired documents.

      USER IMPACT
      Capped collection performance degrades over time. Note that the oplog is a capped collection, so users running replica sets with WiredTiger may be impacted by this issue even if no other capped collections are used.

      RESOLUTION DETAILS
      WiredTiger now caches the current "first" unexpired document in a capped collection. This change improves performance for all capped collections, but is particular important for the performance of replication because the oplog depends on capped collection performance.

      AFFECTED VERSIONS
      MongoDB 3.0.0 through 3.0.4.

      FIX VERSION
      The fix is included in the 3.0.5 production release.

      Original description

      Over the course of 2-3 days, running a simple insert workload with hammer, the performance of inserts degrades from about 20K/s to about 4K/s.
      This was a single node replica set, writing locally to the master, using --nojournal to avoid intertwining any other potential issues.

      Perf indicates this:

      • 99.99% __curfile_next mongo::WiredTigerRecordStore::cappedDeleteAsNeeded_inlock(mongo::OperationContext*, mongo::RecordId const&)
      1. master-oplog-truncates-timeseries.png
        94 kB
      2. patched-oplog-truncates-timeseries.png
        98 kB
      3. sad_oplog.png
        45 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18875 Don't keep deleted pages around during a checkpoint if the checkpoint is the only potential reader. Also, keep internal pages with deleted-but-not-yet-freed children dirty, so deleted leaf pages can't accumulate.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/cefeb2f6449460aff2146c85341438173143e7fa

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18875 Don't keep deleted pages around during a checkpoint if the checkpoint is the only potential reader. Also, keep internal pages with deleted-but-not-yet-freed children dirty, so deleted leaf pages can't accumulate. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/cefeb2f6449460aff2146c85341438173143e7fa
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18875 During a checkpoint, don't keep data alive if the checkpoint is the only reader.

          In particular, if we see pages marked deleted at the beginning of a capped collection, don't cache those pages if there are no possible readers other than the checkpoint itself.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/847262185e60610a91e77e76ace5016c1cb8e736

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18875 During a checkpoint, don't keep data alive if the checkpoint is the only reader. In particular, if we see pages marked deleted at the beginning of a capped collection, don't cache those pages if there are no possible readers other than the checkpoint itself. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/847262185e60610a91e77e76ace5016c1cb8e736
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18875 Track the checkpoint's session ID (rather than the transaction ID): changing visibility broke an assertion when releasing the checkpoint's transaction snapshot.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/35eece43f483f1a715a26556fd1f15a9a0a90ab3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18875 Track the checkpoint's session ID (rather than the transaction ID): changing visibility broke an assertion when releasing the checkpoint's transaction snapshot. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/35eece43f483f1a715a26556fd1f15a9a0a90ab3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2028 from wiredtiger/deleted-leaf-leak

          SERVER-18875 Clean up deleted pages
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/8f7da9ac596ed295228a792383bd5c03da205843

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2028 from wiredtiger/deleted-leaf-leak SERVER-18875 Clean up deleted pages Branch: develop https://github.com/wiredtiger/wiredtiger/commit/8f7da9ac596ed295228a792383bd5c03da205843
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18875 Don't keep deleted pages during a checkpoint. Sync transaction up with WiredTiger 2.6 to ease back-porting, including picking up WT-1912.
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/3751941f5339b257d9fd9c19879f1a901facfbb6

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18875 Don't keep deleted pages during a checkpoint. Sync transaction up with WiredTiger 2.6 to ease back-porting, including picking up WT-1912 . Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/3751941f5339b257d9fd9c19879f1a901facfbb6
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: SERVER-18875: clean up some comments associated with cefeb2f, reorder a
          test to do less work during checkpoint reconciliation.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/1685742d02f4f39f47b5d4a855674fd6dd49f097

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: SERVER-18875 : clean up some comments associated with cefeb2f, reorder a test to do less work during checkpoint reconciliation. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/1685742d02f4f39f47b5d4a855674fd6dd49f097

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile