@michaelcahill, I've stared at this one, but I don't see it.
One thread is looking at page->type:
733 parent_decr += size;
734 WT_TRET(__wt_session_fotxn_add(session, pindex, size));
735 switch (parent->type) {
736 case WT_PAGE_ROW_INT:
737 case WT_PAGE_ROW_LEAF:
738 if ((ikey = __wt_ref_key_instantiated(ref)) == NULL)
739 break;
740 size = sizeof(WT_IKEY) + ikey->size;
741 parent_decr += size;
742 WT_TRET(__wt_session_fotxn_add(session, ikey, size));
743 break;
744 }
and the page has been evicted/freed.
I want to blame the code that loops until we get a matching ref->home and parent, but I don't see any problems there.
We looked at page->type 20 lines before, line 713, so this must have happened simultaneously, so somehow our parent got evicted while we were splitting into it.
Is there some way an internal page could be selected for eviction after a split but before the child pages appear on it? I don't see how that's possible, either.
=================================================================
==105189==ERROR: AddressSanitizer: heap-use-after-free on address 0x607000bc18f0 at pc 0x4febbf bp 0x7f64b0cdbc50 sp 0x7f64b0cdbc48
READ of size 1 at 0x607000bc18f0 thread T79
#0 0x4febbe in __wt_split_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735
WT-1 0x4fb3e3 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:172
WT-2 0x4fb3e3 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88
WT-3 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393
WT-4 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977
WT-5 0x599585 in __wt_cache_full_check <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cache.i>:93
WT-6 0x599585 in __cursor_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:57
WT-7 0x599585 in __curfile_enter <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:94
WT-8 0x599585 in __cursor_func_init <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/cursor.i>:141
WT-9 0x59cc17 in __wt_btcur_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:522:38
WT-10 0x547062 in __curfile_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:262
WT-11 0x452fd7 in row_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:701
WT-12 0x452fd7 in ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:372
WT-13 0x4415b3 in __asan::AsanThread::ThreadStart(unsigned long) (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x4415b3)>
WT-14 0x339e807c52 in start_thread (/lib64/libpthread.so.0+0x339e807c52)
WT-15 0x339e0f5dbc in __clone (/lib64/libc.so.6+0x339e0f5dbc)
0x607000bc18f0 is located 64 bytes inside of 72-byte region [0x607000bc18b0,0x607000bc18f8)
freed by thread T66 here:
#0 0x43af54 in free (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43af54)>
WT-1 0x5a50f3 in __wt_page_out <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_discard.c>:97
WT-2 0x4fb738 in __rec_page_dirty_update <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:192
WT-3 0x4fb738 in __wt_rec_evict <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_evict.c>:88
WT-4 0x4d1bdd in __wt_evict_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:393
WT-5 0x4d1bdd in __wt_evict_lru_page <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_evict.c>:977
previously allocated by thread T78 here:
#0 0x43b0a3 in calloc (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x43b0a3)>
WT-1 0x4889cc in __wt_calloc <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/os_posix/os_alloc.c>:38
WT-2 0x4db993 in __wt_page_inmem <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:294
WT-3 0x4debe6 in __wt_cache_read <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_read.c>:109
WT-4 0x4da997 in __wt_page_in_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_page.c>:46
WT-5 0x52c51e in __wt_page_swap_func <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/include/btree.i>:704
WT-6 0x52c51e in __wt_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/row_srch.c>:278
WT-7 0x59b713 in __cursor_row_search <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:158
WT-8 0x59b713 in __wt_btcur_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/bt_cursor.c>:393
WT-9 0x5464c7 in __curfile_insert <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/cursor/cur_file.c>:223
Thread T79 created by T0 here:
#0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
Thread T66 created by T0 here:
#0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
Thread T78 created by T0 here:
#0 0x436fa0 in __interceptor_pthread_create (<http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/t+0x436fa0)>
WT-1 0x450949 in wts_ops <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/ops.c>:94
WT-2 0x456992 in main <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/test/format/../../../test/format/t.c>:180
SUMMARY: AddressSanitizer: heap-use-after-free <http://mjc.homeunix.org:8180/job/wiredtiger-test-format-stress-santizer/ws/build_posix/../src/btree/rec_split.c>:735 __wt_split_evict
Shadow bytes around the buggy address:
0x0c0e801702c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e801702d0: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
0x0c0e801702e0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
0x0c0e801702f0: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
0x0c0e80170300: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c0e80170310: fd fa fa fa fa fa fd fd fd fd fd fd fd fd[fd]fa
0x0c0e80170320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e80170330: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e80170340: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fd fd
0x0c0e80170350: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
0x0c0e80170360: fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
ASan internal: fe
==105189==ABORTING
+ status=1
+ '[' -f RUNDIR/run ']'
+ cat RUNDIR/run
############################################
# RUN PARAMETERS
############################################
auto_throttle=1
firstfit=1
# bitcnt not applicable to this run
bloom=1
bloom_bit_count=48
bloom_hash_count=7
bloom_oldest=0
cache=3
checkpoints=1
checksum=off
chunk_size=7
compaction=0
compression=none
data_extend=0
data_source=file
delete_pct=12
dictionary=0
file_type=row-store
hot_backups=0
huffman_key=0
huffman_value=0
insert_pct=18
internal_key_truncation=1
internal_page_max=14
key_gap=1
key_max=72
key_min=25
leaf_page_max=10
merge_max=19
merge_threads=1
mmap=1
ops=100000
prefix_compression=0
prefix_compression_min=8
repeat_data_pct=7
reverse=0
rows=100000
runs=1
split_pct=58
statistics=0
threads=30
value_max=814
value_min=19
# wiredtiger_config not applicable to this run
write_pct=59
############################################
- related to
-
WT-1 placeholder WT-1
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
(10 related to)