Sue LoVerso added a comment - Sep 24 2015 01:59:16 PM +00:00 The output from sanitizer is: 59901==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000070 (pc 0x000000a02e22 bp 0x7fdeaeffd7d0 sp 0x7fdeaeffd5c0 T2) #0 0xa02e21 in __wt_addr_string <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_misc.c>:119:2 #1 0xebc6ae in __debug_ref <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_debug.c>:970:6 #2 0xeace0e in __debug_page_row_int <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_debug.c>:815:33 #3 0xea5709 in __debug_page <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_debug.c>:582 #4 0xea32f4 in __debug_tree <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_debug.c>:543:8 #5 0xea2b40 in __wt_debug_tree_all <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_debug.c>:491:10 #6 0x7fdeaeffe6ae (<unknown module>)   AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/btree/bt_misc.c>:119 __wt_addr_string Thread T2 created by T0 here: #0 0x4a230f in pthread_create (<http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/t+0x4a230f)> #1 0x70d5bc in __wt_thread_create <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/os_posix/os_thread.c>:22:70 #2 0x59970f in __wt_sweep_create <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/conn/conn_sweep.c>:381:32 #3 0x587e6c in __wt_connection_workers <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/conn/conn_open.c>:245:32 #4 0x52f611 in wiredtiger_open <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/../src/conn/conn_api.c>:2035:19 #5 0x50de35 in wts_open <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/../../../test/format/wts.c>:238:13 #6 0x503505 in main <http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/ws/build_posix/test/format/../../../test/format/t.c>:197:3 #7 0x31764206ff in __libc_start_main (/lib64/libc.so.6+0x31764206ff) The CONFIG is: ############################################ # RUN PARAMETERS ############################################ abort=0 auto_throttle=0 firstfit=0 bitcnt=2 bloom=1 bloom_bit_count=35 bloom_hash_count=20 bloom_oldest=0 cache=8 checkpoints=1 checksum=uncompressed chunk_size=2 compaction=0 compression=lz4 data_extend=1 data_source=file delete_pct=4 dictionary=0 encryption=none evict_max=5 file_type=row-store backups=0 huffman_key=0 huffman_value=0 insert_pct=28 internal_key_truncation=1 internal_page_max=17 isolation=read-committed key_gap=7 key_max=63 key_min=17 leak_memory=0 leaf_page_max=12 logging=0 logging_compression=zlib logging_archive=0 logging_prealloc=1 long_running_txn=0 lsm_worker_threads=4 merge_max=18 mmap=1 ops=100000 prefix_compression=1 prefix_compression_min=2 repeat_data_pct=80 reverse=0 rows=100000 runs=1 salvage=1 split_pct=52 statistics=0 statistics_server=0 threads=8 timer=20 value_max=753 value_min=3 verify=1 wiredtiger_config= write_pct=88 ############################################

Keith Bostic added a comment - Sep 24 2015 03:19:31 PM +00:00 This one doesn't make any sense to me: wt_debug_tree_all is a debugging routine to dump the in-memory tree, and it's not called from anywhere, let alone from the sweep server. Absent the CONFIG reproducing, I don't see any path to figuring this one out.

Alexander Gorrod added a comment - Sep 24 2015 10:59:02 PM +00:00 This was my fault - I was digging into a cache full hang and triggered the segfault while debugging it.

Githook User added a comment - Sep 24 2015 11:51:55 PM +00:00 Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: WT-2132 Be more defensive in code used to debug address structures. The code is used via diagnostics and salvage. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/bfc8212ae03f008ef7d920a07f9209eb765fdbb6

Githook User added a comment - Sep 24 2015 11:51:56 PM +00:00 Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2217 from wiredtiger/btree-dump-safer WT-2132 Be more defensive in code used to debug address structures. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/435026b180c36ed19012343d1876249f908acb5c

Githook User added a comment - Sep 25 2015 02:25:10 AM +00:00 Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: WT-2132 Make debug dump function more robust to errors. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e084298be2bdb5ed78b1d0e57391e7a13c305dac

Githook User added a comment - Sep 25 2015 02:25:11 AM +00:00 Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2218 from wiredtiger/btree-dump-safer2 WT-2132 Make debug dump function more robust to errors. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/02a3d9fdc76303a61897ee58a8acbcd0bf6c987e