d20006| ==3317== ERROR: AddressSanitizer: heap-use-after-free on address 0x6004002663b8 at pc 0x3f0af42 bp 0x7fc0e283d190 sp 0x7fc0e283d188 d20006| READ of size 8 at 0x6004002663b8 thread T80 d20006| 2015-11-17T16:18:16.015-0500 D STORAGE [repl writer worker 15] WT queued drop of table:collection-11-6531197468317506066 res 16 d20006| 2015-11-17T16:18:16.015-0500 D STORAGE [repl writer worker 15] WT queued drop of table:index-12-6531197468317506066 res 16 d20006| 2015-11-17T16:18:16.016-0500 D STORAGE [repl writer worker 15] WT queued drop of table:index-13-6531197468317506066 res 16 d20006| 2015-11-17T16:18:16.016-0500 D STORAGE [repl writer worker 15] WT queued drop of table:index-14-6531197468317506066 res 16 d20006| 2015-11-17T16:18:16.016-0500 D STORAGE [repl writer worker 15] WT queued drop of table:index-15-6531197468317506066 res 16 d20006| #0 0x3f0af41 in __split_parent /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1191 d20006| #1 0x3f0e585 in __split_multi /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1658 d20006| #2 0x3f0e585 in __wt_split_multi /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1701 d20006| #3 0x3fda6a2 in __evict_page_dirty_update /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:285 d20006| #4 0x3fda6a2 in __wt_evict /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:123 d20006| #5 0x3fd08c9 in __evict_page /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:1467 d20006| #6 0x3fd14b9 in __evict_lru_pages /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:818 d20006| #7 0x3fd14b9 in __evict_worker /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:434 d20006| #8 0x7fc118696b97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97) d20006| #9 0x7fc11783c181 in start_thread /build/buildd/eglibc-2.19/nptl/pthread_create.c:312 d20006| #10 0x7fc11756947c in clone /build/buildd/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111 d20006| 0x6004002663b8 is located 8 bytes inside of 16-byte region [0x6004002663b0,0x6004002663c0) d20006| freed by thread T27 here: d20006| #0 0x7fc11869333a in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x1533a) d20006| #1 0x3f088a5 in __split_parent /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1193 d20006| #2 0x3f0d084 in __split_insert /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1493 d20006| #3 0x3f0d084 in __wt_split_insert /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:1547 d20006| #4 0x3fda9ad in __evict_review /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:430 d20006| #5 0x3fda9ad in __wt_evict /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_page.c:79 d20006| #6 0x3ef46c3 in __wt_page_release_evict /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/include/btree.i:1164 d20006| #7 0x3ef46c3 in __wt_page_in_func /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_read.c:536 d20006| #8 0x3f414cf in __wt_page_swap_func /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/include/btree.i:1260 d20006| #9 0x3f414cf in __wt_row_search /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/row_srch.c:310 d20006| #10 0x3eba0cf in __cursor_row_search /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_cursor.c:241 d20006| #11 0x3eba0cf in __wt_btcur_insert /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_cursor.c:534 d20006| #12 0x3f90ca9 in __curfile_insert /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/cursor/cur_file.c:245 d20006| #13 0x20969f0 in mongo::WiredTigerRecordStore::insertRecords(mongo::OperationContext*, std::vector >*, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:1243 d20006| #14 0xe3d690 in mongo::Collection::_insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/catalog/collection.cpp:428 d20006| #15 0xe3e4dc in mongo::Collection::insertDocuments(mongo::OperationContext*, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >, bool, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/catalog/collection.cpp:358 d20006| #16 0x1ab257e in mongo::repl::writeOpsToOplog(mongo::OperationContext*, std::vector > const&) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/oplog.cpp:487 d20006| #17 0x1caab9f in mongo::repl::SyncTail::multiApply(mongo::OperationContext*, mongo::repl::SyncTail::OpQueue const&) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/sync_tail.cpp:477 d20006| #18 0x1cb0c1a in mongo::repl::SyncTail::oplogApplication() /home/grund/MongoDB/replcrsh/src/mongo/db/repl/sync_tail.cpp:697 d20006| #19 0x1c9353a in mongo::repl::runSyncThread() /home/grund/MongoDB/replcrsh/src/mongo/db/repl/rs_sync.cpp:133 d20006| #20 0x7fc11801fa3f (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb1a3f) d20006| previously allocated by thread T28 here: d20006| #0 0x7fc1186934e5 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x154e5) d20006| #1 0x401f9a1 in __wt_calloc /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/os_posix/os_alloc.c:60 d20006| #2 0x3ed96df in __wt_delete_page /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_delete.c:135 d20006| #3 0x3f2a036 in __wt_tree_walk /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_walk.c:270 d20006| #4 0x3e917aa in __wt_btcur_next /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_curnext.c:530 d20006| #5 0x3ecc1bf in __cursor_truncate /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_cursor.c:1009 d20006| #6 0x3ed7236 in __wt_btcur_range_truncate /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_cursor.c:1137 d20006| #7 0x40975f0 in __wt_schema_range_truncate /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/schema/schema_truncate.c:178 d20006| #8 0x40a2ba4 in __session_truncate /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/session/session_api.c:769 d20006| #9 0x209e470 in mongo::WiredTigerRecordStore::temp_cappedTruncateAfter(mongo::OperationContext*, mongo::RecordId, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp:1716 d20006| #10 0xe36392 in mongo::Collection::temp_cappedTruncateAfter(mongo::OperationContext*, mongo::RecordId, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/catalog/collection.cpp:790 d20006| #11 0x1c8b6b6 in syncFixUp /home/grund/MongoDB/replcrsh/src/mongo/db/repl/rs_rollback.cpp:760 d20006| #12 0x1c8b6b6 in _syncRollback /home/grund/MongoDB/replcrsh/src/mongo/db/repl/rs_rollback.cpp:848 d20006| #13 0x1c8b6b6 in mongo::repl::syncRollback(mongo::OperationContext*, mongo::repl::OpTime const&, mongo::repl::OplogInterface const&, mongo::repl::RollbackSource const&, mongo::repl::ReplicationCoordinator*, std::function >)> const&) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/rs_rollback.cpp:907 d20006| #14 0x1c92511 in mongo::repl::syncRollback(mongo::OperationContext*, mongo::repl::OpTime const&, mongo::repl::OplogInterface const&, mongo::repl::RollbackSource const&, mongo::repl::ReplicationCoordinator*) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/rs_rollback.cpp:923 d20006| #15 0x1996f04 in mongo::repl::BackgroundSync::_rollback(mongo::OperationContext*, mongo::HostAndPort const&, std::function) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/bgsync.cpp:614 d20006| #16 0x19a102a in mongo::repl::BackgroundSync::_produce(mongo::OperationContext*) /home/grund/MongoDB/replcrsh/src/mongo/db/repl/bgsync.cpp:394 d20006| #17 0x19a23e0 in mongo::repl::BackgroundSync::_producerThread() /home/grund/MongoDB/replcrsh/src/mongo/db/repl/bgsync.cpp:250 d20006| #18 0x19a31f4 in mongo::repl::BackgroundSync::producerThread() /home/grund/MongoDB/replcrsh/src/mongo/db/repl/bgsync.cpp:196 d20006| #19 0x7fc11801fa3f (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb1a3f) d20006| Thread T80 created by T6 here: d20006| #0 0x7fc118688b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b) d20006| #1 0x40281c9 in __wt_thread_create /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/os_posix/os_thread.c:22 d20006| #2 0x3fd33f5 in __evict_pass /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:580 d20006| #3 0x3fd33f5 in __evict_server /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:172 d20006| #4 0x7fc118696b97 (/usr/lib/x86_64-linux-gnu/libasan.so.0+0x18b97) d20006| Thread T6 created by T0 here: d20006| #0 0x7fc118688b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b) d20006| #1 0x40281c9 in __wt_thread_create /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/os_posix/os_thread.c:22 d20006| #2 0x3fd694f in __wt_evict_create /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/evict/evict_lru.c:343 d20006| #3 0x3f73256 in __wt_connection_workers /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/conn/conn_open.c:268 d20006| #4 0x3f60be6 in wiredtiger_open /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/conn/conn_api.c:2061 d20006| #5 0x20855d3 in mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::string const&, std::string const&, bool, bool, bool) /home/grund/MongoDB/replcrsh/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:198 d20006| #6 0x207db9f in mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const&) const /home/grund/MongoDB/replcrsh/src/mongo/db/storage/wiredtiger/wiredtiger_init.cpp:68 d20006| #7 0x1dd1c71 in mongo::ServiceContextMongoD::initializeGlobalStorageEngine() /home/grund/MongoDB/replcrsh/src/mongo/db/service_context_d.cpp:148 d20006| #8 0xa35844 in _initAndListen /home/grund/MongoDB/replcrsh/src/mongo/db/db.cpp:463 d20006| #9 0xa35844 in mongo::initAndListen(int) /home/grund/MongoDB/replcrsh/src/mongo/db/db.cpp:644 d20006| #10 0x99c9c6 in mongoDbMain /home/grund/MongoDB/replcrsh/src/mongo/db/db.cpp:878 d20006| #11 0x99c9c6 in main /home/grund/MongoDB/replcrsh/src/mongo/db/db.cpp:689 d20006| #12 0x7fc117490ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287 d20006| Thread T27 created by T13 here: d20006| #0 0x7fc118688b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b) d20006| #1 0x7fc11801fc8e in std::thread::_M_start_thread(std::shared_ptr) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb1c8e) d20006| Thread T13 created by T0 here: d20006| #0 0x7fc118688b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b) d20006| #1 0x7fc11801fc8e in std::thread::_M_start_thread(std::shared_ptr) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb1c8e) d20006| #2 0x60580000d87f (+0xd87f) d20006| Thread T28 created by T13 here: d20006| #0 0x7fc118688b5b in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.0+0xab5b) d20006| #1 0x7fc11801fc8e in std::thread::_M_start_thread(std::shared_ptr) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb1c8e) d20006| SUMMARY: AddressSanitizer: heap-use-after-free /home/grund/MongoDB/replcrsh/src/third_party/wiredtiger/src/btree/bt_split.c:205 __split_should_deepen d20006| Shadow bytes around the buggy address: d20006| 0x0c0100044c20: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| 0x0c0100044c30: fa fa fd fd fa fa 00 00 fa fa fd fd fa fa fd fd d20006| 0x0c0100044c40: fa fa fd fd fa fa fd fd fa fa 00 00 fa fa 00 00 d20006| 0x0c0100044c50: fa fa 00 00 fa fa 00 00 fa fa fd fd fa fa fd fd d20006| 0x0c0100044c60: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| =>0x0c0100044c70: fa fa fd fd fa fa fd[fd]fa fa fd fd fa fa fd fd d20006| 0x0c0100044c80: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| 0x0c0100044c90: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| 0x0c0100044ca0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| 0x0c0100044cb0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| 0x0c0100044cc0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd d20006| Shadow byte legend (one shadow byte represents 8 application bytes): d20006| Addressable: 00 d20006| Partially addressable: 01 02 03 04 05 06 07 d20006| Heap left redzone: fa d20006| Heap righ redzone: fb d20006| Freed Heap region: fd d20006| Stack left redzone: f1 d20006| Stack mid redzone: f2 d20006| Stack right redzone: f3 d20006| Stack partial redzone: f4 d20006| Stack after return: f5 d20006| Stack use after scope: f8 d20006| Global redzone: f9 d20006| Global init order: f6 d20006| Poisoned by user: f7 d20006| ASan internal: fe d20006| ==3317== ABORTING