Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18321

Speed up background index build with WiredTiger LSM

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • 3.0.5, 3.1.3
    • Affects Version/s: None
    • Component/s: WiredTiger
    • None
    • Fully Compatible

      I noticed that the repl13.js test is very slow with WiredTiger and LSM. While digging into the performance, I saw that the slow part of the test was building an index on the secondary.

      It appears as though each entry is inserted with a WiredTiger auto-commit transaction, each auto-commit transaction is doing a write and fsync to the WiredTiger WAL (journal). The index being built is 100,000 entries - so 100k fsync calls.

      A representative call stack is:

      (gdb) where
      #0  0x00007fecbb6d8c4d in fdatasync () from /lib64/libc.so.6
      #1  0x0000000001336e7c in __wt_handle_sync (fd=18)
          at src/third_party/wiredtiger/src/os_posix/os_fsync.c:44
      #2  0x0000000001337192 in __wt_fsync (session=session@entry=0x3a8a040, fh=0x3657750)
          at src/third_party/wiredtiger/src/os_posix/os_fsync.c:129
      #3  0x0000000001321a1a in __log_release (session=session@entry=0x3a8a040, 
          slot=slot@entry=0x7fecae8c5a00, freep=freep@entry=0x7fecae8c59ec)
          at src/third_party/wiredtiger/src/log/log.c:1043
      #4  0x0000000001322b28 in __log_direct_write (session=session@entry=0x3a8a040, 
          record=record@entry=0x3a3f410, lsnp=lsnp@entry=0x0, flags=flags@entry=4)
          at src/third_party/wiredtiger/src/log/log.c:1542
      #5  0x0000000001324139 in __log_write_internal (flags=4, lsnp=0x0, record=0x3a3f410, 
          session=0x3a8a040) at src/third_party/wiredtiger/src/log/log.c:1707
      #6  __wt_log_write (session=session@entry=0x3a8a040, record=<optimized out>, lsnp=lsnp@entry=0x0, 
          flags=4) at src/third_party/wiredtiger/src/log/log.c:1649
      #7  0x000000000136ca84 in __wt_txn_log_commit (session=session@entry=0x3a8a040, cfg=cfg@entry=0x0)
          at src/third_party/wiredtiger/src/txn/txn_log.c:209
      #8  0x0000000001369b10 in __wt_txn_commit (session=0x3a8a040, cfg=0x0)
          at src/third_party/wiredtiger/src/txn/txn.c:419
      #9  0x000000000132b785 in __clsm_insert (cursor=0x3915ee0)
          at src/third_party/wiredtiger/src/lsm/lsm_cursor.c:1361
      #10 0x0000000000d71298 in mongo::WiredTigerIndex::UniqueBulkBuilder::doInsert (this=0x50d6340)
          at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:582
      #11 0x0000000000d71945 in mongo::WiredTigerIndex::UniqueBulkBuilder::addKey (this=0x50d6340, 
          newKey=..., loc=...) at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:531
      #12 0x0000000000a92fb7 in mongo::IndexAccessMethod::commitBulk (this=0x5b1bd40, 
          txn=0x7fecae8c78f0, 
          bulk=std::unique_ptr<mongo::IndexAccessMethod::BulkBuilder> containing 0x5b11e80, 
          mayInterrupt=false, dupsAllowed=false, dupsToDrop=dupsToDrop@entry=0x7fecae8c6810)
          at src/mongo/db/index/index_access_method.cpp:410
      #13 0x0000000000949b28 in mongo::MultiIndexBlock::doneInserting (this=this@entry=0x7fecae8c6840, 
          dupsOut=dupsOut@entry=0x7fecae8c6810) at src/mongo/db/catalog/index_create.cpp:371
      #14 0x000000000094a130 in mongo::MultiIndexBlock::insertAllDocumentsInCollection (
          this=this@entry=0x7fecae8c6840, dupsOut=dupsOut@entry=0x7fecae8c6810)
          at src/mongo/db/catalog/index_create.cpp:328
      #15 0x0000000000955b13 in mongo::Cloner::go (this=this@entry=0x7fecae8c6940, 
          txn=txn@entry=0x7fecae8c78f0, toDBName="d", masterHost="127.0.0.1:31000", opts=..., 
          clonedColls=clonedColls@entry=0x0, errmsg="", errCode=errCode@entry=0x7fecae8c690c)
          at src/mongo/db/cloner.cpp:628
      #16 0x0000000000c0cf46 in mongo::repl::ReplSource::resync (this=this@entry=0x3a0e3c0, 
          txn=txn@entry=0x7fecae8c78f0, dbName=...) at src/mongo/db/repl/master_slave.cpp:492
      #17 0x0000000000c108c8 in mongo::repl::ReplSource::_sync_pullOpLog_applyOperation (
          this=this@entry=0x3a0e3c0, txn=txn@entry=0x7fecae8c78f0, op=..., 
          alreadyLocked=alreadyLocked@entry=false) at src/mongo/db/repl/master_slave.cpp:765
      

      Ideally doing such an index build in WiredTiger would turn on bulk loading, or wrap groups of transactions into commits or in the least doing non-sync transactional operations.

      The obvious place to make that change is in mongo::MultiIndexBlock::insertAllDocumentsInCollection, but that's outside the WiredTiger storage engine implementation layer, so I'm hesitant to jump into the change without some discussion.

      There is already a call to WiredTigerRecoveryUnit::setRollbackWritesDisabled in IndexAccessMethod::commitBulk, but it isn't obvious how to use that information to help from the WiredTiger storage engine side.

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: