-
Type:
Improvement
-
Resolution: Done
-
Priority:
Minor - P4
-
Affects Version/s: None
-
Component/s: WiredTiger
-
Labels:None
-
Fully Compatible
I noticed that the repl13.js test is very slow with WiredTiger and LSM. While digging into the performance, I saw that the slow part of the test was building an index on the secondary.
It appears as though each entry is inserted with a WiredTiger auto-commit transaction, each auto-commit transaction is doing a write and fsync to the WiredTiger WAL (journal). The index being built is 100,000 entries - so 100k fsync calls.
A representative call stack is:
(gdb) where
#0 0x00007fecbb6d8c4d in fdatasync () from /lib64/libc.so.6
#1 0x0000000001336e7c in __wt_handle_sync (fd=18)
at src/third_party/wiredtiger/src/os_posix/os_fsync.c:44
#2 0x0000000001337192 in __wt_fsync (session=session@entry=0x3a8a040, fh=0x3657750)
at src/third_party/wiredtiger/src/os_posix/os_fsync.c:129
#3 0x0000000001321a1a in __log_release (session=session@entry=0x3a8a040,
slot=slot@entry=0x7fecae8c5a00, freep=freep@entry=0x7fecae8c59ec)
at src/third_party/wiredtiger/src/log/log.c:1043
#4 0x0000000001322b28 in __log_direct_write (session=session@entry=0x3a8a040,
record=record@entry=0x3a3f410, lsnp=lsnp@entry=0x0, flags=flags@entry=4)
at src/third_party/wiredtiger/src/log/log.c:1542
#5 0x0000000001324139 in __log_write_internal (flags=4, lsnp=0x0, record=0x3a3f410,
session=0x3a8a040) at src/third_party/wiredtiger/src/log/log.c:1707
#6 __wt_log_write (session=session@entry=0x3a8a040, record=<optimized out>, lsnp=lsnp@entry=0x0,
flags=4) at src/third_party/wiredtiger/src/log/log.c:1649
#7 0x000000000136ca84 in __wt_txn_log_commit (session=session@entry=0x3a8a040, cfg=cfg@entry=0x0)
at src/third_party/wiredtiger/src/txn/txn_log.c:209
#8 0x0000000001369b10 in __wt_txn_commit (session=0x3a8a040, cfg=0x0)
at src/third_party/wiredtiger/src/txn/txn.c:419
#9 0x000000000132b785 in __clsm_insert (cursor=0x3915ee0)
at src/third_party/wiredtiger/src/lsm/lsm_cursor.c:1361
#10 0x0000000000d71298 in mongo::WiredTigerIndex::UniqueBulkBuilder::doInsert (this=0x50d6340)
at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:582
#11 0x0000000000d71945 in mongo::WiredTigerIndex::UniqueBulkBuilder::addKey (this=0x50d6340,
newKey=..., loc=...) at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp:531
#12 0x0000000000a92fb7 in mongo::IndexAccessMethod::commitBulk (this=0x5b1bd40,
txn=0x7fecae8c78f0,
bulk=std::unique_ptr<mongo::IndexAccessMethod::BulkBuilder> containing 0x5b11e80,
mayInterrupt=false, dupsAllowed=false, dupsToDrop=dupsToDrop@entry=0x7fecae8c6810)
at src/mongo/db/index/index_access_method.cpp:410
#13 0x0000000000949b28 in mongo::MultiIndexBlock::doneInserting (this=this@entry=0x7fecae8c6840,
dupsOut=dupsOut@entry=0x7fecae8c6810) at src/mongo/db/catalog/index_create.cpp:371
#14 0x000000000094a130 in mongo::MultiIndexBlock::insertAllDocumentsInCollection (
this=this@entry=0x7fecae8c6840, dupsOut=dupsOut@entry=0x7fecae8c6810)
at src/mongo/db/catalog/index_create.cpp:328
#15 0x0000000000955b13 in mongo::Cloner::go (this=this@entry=0x7fecae8c6940,
txn=txn@entry=0x7fecae8c78f0, toDBName="d", masterHost="127.0.0.1:31000", opts=...,
clonedColls=clonedColls@entry=0x0, errmsg="", errCode=errCode@entry=0x7fecae8c690c)
at src/mongo/db/cloner.cpp:628
#16 0x0000000000c0cf46 in mongo::repl::ReplSource::resync (this=this@entry=0x3a0e3c0,
txn=txn@entry=0x7fecae8c78f0, dbName=...) at src/mongo/db/repl/master_slave.cpp:492
#17 0x0000000000c108c8 in mongo::repl::ReplSource::_sync_pullOpLog_applyOperation (
this=this@entry=0x3a0e3c0, txn=txn@entry=0x7fecae8c78f0, op=...,
alreadyLocked=alreadyLocked@entry=false) at src/mongo/db/repl/master_slave.cpp:765
Ideally doing such an index build in WiredTiger would turn on bulk loading, or wrap groups of transactions into commits or in the least doing non-sync transactional operations.
The obvious place to make that change is in mongo::MultiIndexBlock::insertAllDocumentsInCollection, but that's outside the WiredTiger storage engine implementation layer, so I'm hesitant to jump into the change without some discussion.
There is already a call to WiredTigerRecoveryUnit::setRollbackWritesDisabled in IndexAccessMethod::commitBulk, but it isn't obvious how to use that information to help from the WiredTiger storage engine side.
- depends on
-
-
- Closed
-