Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18321

Speed up background index build with WiredTiger LSM

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.5, 3.1.3
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Backport Completed:

      Description

      I noticed that the repl13.js test is very slow with WiredTiger and LSM. While digging into the performance, I saw that the slow part of the test was building an index on the secondary.

      It appears as though each entry is inserted with a WiredTiger auto-commit transaction, each auto-commit transaction is doing a write and fsync to the WiredTiger WAL (journal). The index being built is 100,000 entries - so 100k fsync calls.

      A representative call stack is:

      (gdb) where
      #0  0x00007fecbb6d8c4d in fdatasync () from /lib64/libc.so.6
      #1  0x0000000001336e7c in __wt_handle_sync (fd=18)
          at src/third_party/wiredtiger/src/os_posix/os_fsync.c:44
      #2  0x0000000001337192 in __wt_fsync (session=session@entry=0x3a8a040, fh=0x3657750)
          at src/third_party/wiredtiger/src/os_posix/os_fsync.c:129
      #3  0x0000000001321a1a in __log_release (session=session@entry=0x3a8a040, 
          slot=slot@entry=0x7fecae8c5a00, freep=freep@entry=0x7fecae8c59ec)
          at src/third_party/wiredtiger/src/log/log.c:1043
      #4  0x0000000001322b28 in __log_direct_write (session=session@entry=0x3a8a040, 
          record=record@entry=0x3a3f410, lsnp=lsnp@entry=0x0, flags=flags@entry=4)
          at src/third_party/wiredtiger/src/log/log.c:1542
      #5  0x0000000001324139 in __log_write_internal (flags=4, lsnp=0x0, record=0x3a3f410, 
          session=0x3a8a040) at src/third_party/wiredtiger/src/log/log.c:1707
      #6  __wt_log_write (session=session@entry=0x3a8a040, record=<optimized out>, lsnp=lsnp@entry=0x0, 
          flags=4) at src/third_party/wiredtiger/src/log/log.c:1649
      #7  0x000000000136ca84 in __wt_txn_log_commit (session=session@entry=0x3a8a040, cfg=cfg@entry=0x0)
          at src/third_party/wiredtiger/src/txn/txn_log.c:209
      #8  0x0000000001369b10 in __wt_txn_commit (session=0x3a8a040, cfg=0x0)
          at src/third_party/wiredtiger/src/txn/txn.c:419
      #9  0x000000000132b785 in __clsm_insert (cursor=0x3915ee0)
          at src/third_party/wiredtiger/src/lsm/lsm_cursor.c:1361
      #10 0x0000000000d71298 in mongo::WiredTigerIndex::UniqueBulkBuilder::doInsert (this=0x50d6340)
          at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:582
      #11 0x0000000000d71945 in mongo::WiredTigerIndex::UniqueBulkBuilder::addKey (this=0x50d6340, 
          newKey=..., loc=...) at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:531
      #12 0x0000000000a92fb7 in mongo::IndexAccessMethod::commitBulk (this=0x5b1bd40, 
          txn=0x7fecae8c78f0, 
          bulk=std::unique_ptr<mongo::IndexAccessMethod::BulkBuilder> containing 0x5b11e80, 
          mayInterrupt=false, dupsAllowed=false, dupsToDrop=dupsToDrop@entry=0x7fecae8c6810)
          at src/mongo/db/index/index_access_method.cpp:410
      #13 0x0000000000949b28 in mongo::MultiIndexBlock::doneInserting (this=this@entry=0x7fecae8c6840, 
          dupsOut=dupsOut@entry=0x7fecae8c6810) at src/mongo/db/catalog/index_create.cpp:371
      #14 0x000000000094a130 in mongo::MultiIndexBlock::insertAllDocumentsInCollection (
          this=this@entry=0x7fecae8c6840, dupsOut=dupsOut@entry=0x7fecae8c6810)
          at src/mongo/db/catalog/index_create.cpp:328
      #15 0x0000000000955b13 in mongo::Cloner::go (this=this@entry=0x7fecae8c6940, 
          txn=txn@entry=0x7fecae8c78f0, toDBName="d", masterHost="127.0.0.1:31000", opts=..., 
          clonedColls=clonedColls@entry=0x0, errmsg="", errCode=errCode@entry=0x7fecae8c690c)
          at src/mongo/db/cloner.cpp:628
      #16 0x0000000000c0cf46 in mongo::repl::ReplSource::resync (this=this@entry=0x3a0e3c0, 
          txn=txn@entry=0x7fecae8c78f0, dbName=...) at src/mongo/db/repl/master_slave.cpp:492
      #17 0x0000000000c108c8 in mongo::repl::ReplSource::_sync_pullOpLog_applyOperation (
          this=this@entry=0x3a0e3c0, txn=txn@entry=0x7fecae8c78f0, op=..., 
          alreadyLocked=alreadyLocked@entry=false) at src/mongo/db/repl/master_slave.cpp:765
      

      Ideally doing such an index build in WiredTiger would turn on bulk loading, or wrap groups of transactions into commits or in the least doing non-sync transactional operations.

      The obvious place to make that change is in mongo::MultiIndexBlock::insertAllDocumentsInCollection, but that's outside the WiredTiger storage engine implementation layer, so I'm hesitant to jump into the change without some discussion.

      There is already a call to WiredTigerRecoveryUnit::setRollbackWritesDisabled in IndexAccessMethod::commitBulk, but it isn't obvious how to use that information to help from the WiredTiger storage engine side.

        Issue Links

          Activity

          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          We've realized that this performance problem is due to MongoDB using a bulk cursor for the load. In LSM the bulk flag is ignored to cursor open, thus we end up with auto commit transaction semantics.

          I'll think about whether it makes sense to alter LSM behavior for bulk cursors, or to make changes in the MongoDB layer for LSM indexes.

          Show
          alexander.gorrod Alexander Gorrod added a comment - We've realized that this performance problem is due to MongoDB using a bulk cursor for the load. In LSM the bulk flag is ignored to cursor open, thus we end up with auto commit transaction semantics. I'll think about whether it makes sense to alter LSM behavior for bulk cursors, or to make changes in the MongoDB layer for LSM indexes.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: Add support for bulk load in LSM trees.

          This allows us to load into a single btree, using btree bulk load
          semantics (single threaded, in order, no logging). Once the load completes
          we switch the chunk out for the LSM tree.

          It's possible that we could avoid some of the switch logic when closing a
          bulk load cursor - since the file is flushed when closing the btree handle.
          It's simpler to use the switch logic to update the state of the tree.

          Refs WT-1922 SERVER-18321
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/c82ed17fd2c47d87e525bcd37e4a69c11d0336fe

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: Add support for bulk load in LSM trees. This allows us to load into a single btree, using btree bulk load semantics (single threaded, in order, no logging). Once the load completes we switch the chunk out for the LSM tree. It's possible that we could avoid some of the switch logic when closing a bulk load cursor - since the file is flushed when closing the btree handle. It's simpler to use the switch logic to update the state of the tree. Refs WT-1922 SERVER-18321 Branch: develop https://github.com/wiredtiger/wiredtiger/commit/c82ed17fd2c47d87e525bcd37e4a69c11d0336fe
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #1948 from wiredtiger/lsm-bulk-load

          Add support for bulk load in LSM trees.
          Refs WT-1922 SERVER-18321
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/4d37a27896872dc5d280f5e85666e1d8431ec33b

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #1948 from wiredtiger/lsm-bulk-load Add support for bulk load in LSM trees. Refs WT-1922 SERVER-18321 Branch: develop https://github.com/wiredtiger/wiredtiger/commit/4d37a27896872dc5d280f5e85666e1d8431ec33b
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: WT-1922 Add support for bulk load in LSM trees. Also references
          SERVER-18321

          (cherry picked from commit 4d37a27896872dc5d280f5e85666e1d8431ec33b)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/10eb756c7bb8cc1a6847a2f2fec5fcb2ee883d91

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: WT-1922 Add support for bulk load in LSM trees. Also references SERVER-18321 (cherry picked from commit 4d37a27896872dc5d280f5e85666e1d8431ec33b) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/10eb756c7bb8cc1a6847a2f2fec5fcb2ee883d91

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: