Major - P3 The configuration was column-store, but there is not (yet) strong evidence that the failure is column-store specific.
http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/17291/console
==32018==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000110a10 at pc 0x0000007c9089 bp 0x7fe3e6a07850 sp 0x7fe3e6a07848
WRITE of size 8 at 0x604000110a10 thread T28
#0 0x7c9088 in __wt_page_inmem /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_page.c:233:13
#1 0x7dc19d in __page_read /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:370:2
#2 0x7d9899 in __wt_page_in_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:506:4
#3 0x869be7 in __wt_page_swap_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree.i:1429:8
#4 0x867b3d in __wt_col_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/col_srch.c:194:14
#5 0xa03883 in __cursor_col_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:336:2
#6 0xa0242a in __wt_btcur_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:479:3
#7 0xa0a68f in __wt_btcur_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:1251:2
#8 0x8db542 in __curfile_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:303:2
#9 0x52b704 in col_modify /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:1344:16
#10 0x526795 in ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:769:11
#11 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
#12 0x7fe405a4b76c in __clone (/lib64/libc.so.6+0xf776c)
0x604000110a10 is located 0 bytes inside of 48-byte region [0x604000110a10,0x604000110a40)
freed by thread T1 here:
#0 0x4df570 in __interceptor_free /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:47
#1 0x608329 in __wt_free_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:327:2
#2 0x79f909 in __wt_free_ref /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:331:2
#3 0x79ffea in __wt_free_ref_index /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:348:3
#4 0x7a1bfe in __free_page_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:268:2
#5 0x79ee1c in __page_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:143:3
#6 0x79e0be in __wt_ref_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:60:2
#7 0x79f15e in __wt_ref_out /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:70:2
#8 0x5cf89e in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:378:5
#9 0x5cc720 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:184:3
#10 0x5b229f in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2242:2
#11 0x5ac9d7 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1149:14
#12 0x5abe65 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:318:3
#13 0x7297c3 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:31:3
#14 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
previously allocated by thread T2 here:
#0 0x4dfa6d in calloc /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74
#1 0x606b82 in __wt_calloc /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:52:11
#2 0x811934 in __wt_multi_to_ref /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:1593:2
#3 0x82d510 in __split_multi /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2092:3
#4 0x8143c4 in __split_multi_lock /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2139:13
#5 0x814259 in __wt_split_multi /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_split.c:2167:2
#6 0x5cf3b5 in __evict_page_dirty_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:341:4
#7 0x5cc720 in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:184:3
#8 0x5b229f in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2242:2
#9 0x5ac9d7 in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1149:14
#10 0x5abe65 in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:318:3
#11 0x7297c3 in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:31:3
#12 0x7fe406867dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
Thread T28 created by T0 here:
#0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
#1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
#2 0x5224cb in wts_ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:164:3
#3 0x532fcd in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:209:5
#4 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T1 created by T0 here:
#0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
#1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
#2 0x7273d5 in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:213:3
#3 0x727df0 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:304:2
#4 0x5add95 in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:477:2
#5 0x5773ec in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_open.c:238:2
#6 0x551fb0 in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_api.c:2627:2
#7 0x53d05b in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:273:2
#8 0x532e90 in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:190:3
#9 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
Thread T2 created by T0 here:
#0 0x4373c1 in __interceptor_pthread_create /home/bostic/src/llvm40/projects/compiler-rt/lib/asan/asan_interceptors.cc:305
#1 0x61d2c0 in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:30:2
#2 0x7273d5 in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:213:3
#3 0x727df0 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:304:2
#4 0x5add95 in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:477:2
#5 0x5773ec in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_open.c:238:2
#6 0x551fb0 in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_api.c:2627:2
#7 0x53d05b in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:273:2
#8 0x532e90 in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:190:3
#9 0x7fe405975b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
SUMMARY: AddressSanitizer: heap-use-after-free /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_page.c:233:13 in __wt_page_inmem
Shadow bytes around the buggy address:
0x0c088001a0f0: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
0x0c088001a100: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
0x0c088001a110: fa fa 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
0x0c088001a120: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
0x0c088001a130: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
=>0x0c088001a140: fa fa[fd]fd fd fd fd fd fa fa 00 00 00 00 00 00
0x0c088001a150: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
0x0c088001a160: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c088001a170: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
0x0c088001a180: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 00
0x0c088001a190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==32018==ABORTING
Here's the CONFIG:
############################################ # RUN PARAMETERS ############################################ abort=0 alter=0 auto_throttle=1 backups=0 bitcnt=5 bloom=1 bloom_bit_count=9 bloom_hash_count=12 bloom_oldest=0 cache=36 checkpoints=1 checksum=uncompressed chunk_size=8 compaction=0 compression=lz4 data_extend=0 data_source=table delete_pct=7 dictionary=0 direct_io=0 encryption=rotn-7 evict_max=5 file_type=variable-length column-store firstfit=0 huffman_key=0 huffman_value=0 independent_thread_rng=1 in_memory=0 insert_pct=12 internal_key_truncation=1 internal_page_max=17 isolation=random key_gap=3 key_max=32 key_min=10 leaf_page_max=16 leak_memory=0 logging=0 logging_archive=1 logging_compression=none logging_prealloc=0 long_running_txn=0 lsm_worker_threads=3 merge_max=14 mmap=1 modify_pct=39 ops=100000 prefix_compression=0 prefix_compression_min=7 quiet=1 read_pct=23 rebalance=1 repeat_data_pct=60 reverse=0 rows=100000 runs=1 salvage=1 split_pct=58 statistics=0 statistics_server=0 threads=9 timer=360 transaction_timestamps=0 transaction-frequency=45 value_max=3975 value_min=15 verify=1 wiredtiger_config= write_pct=19 ############################################