-
Type:
Build Failure
-
Resolution: Won't Fix
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Concurrency, Test Format
format-stress-test-disagg-leader-tsan on amazon2023-arm64-tsan
Host: i-03328233d2af4985a
Project: wiredtiger
Commit: 57b822fe
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-leader-tsan task_log
Logs:
WARNING: ThreadSanitizer: data race (pid=7709)
Atomic write of size 8 at 0xffffa0303d18 by main thread (mutexes: write M0):
#0 __wt_atomic_store_double_relaxed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 (libwiredtiger.so.12.0.0+0x2bada4)
#1 __checkpoint_update_evict_triggers_end /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:599:17 (libwiredtiger.so.12.0.0+0x2ba35c)
#2 __checkpoint_db_internal /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1739:5 (libwiredtiger.so.12.0.0+0x2b47e8)
#3 __checkpoint_db_wrapper /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1880:11 (libwiredtiger.so.12.0.0+0x2ac834)
#4 __wt_checkpoint_db /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1959:9 (libwiredtiger.so.12.0.0+0x2abf20)
#5 __wt_txn_global_shutdown /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/txn/txn.c:2559:17 (libwiredtiger.so.12.0.0+0x786a18)
#6 __conn_close /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:1261:5 (libwiredtiger.so.12.0.0+0x2e5c50)
#7 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:763:9 (t+0x1419a0)
#8 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
Previous read of size 8 at 0xffffa0303d18 by thread T2 (mutexes: write M1):
#0 __wti_evict_updates_needed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/../evict/evict_inline.h:532:55 (libwiredtiger.so.12.0.0+0x4782b0)
#1 __evict_update_work /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:765:9 (libwiredtiger.so.12.0.0+0x47b578)
#2 __evict_pass /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:898:9 (libwiredtiger.so.12.0.0+0x479920)
#3 __evict_server /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:463:5 (libwiredtiger.so.12.0.0+0x478c44)
#4 __evict_thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:331:15 (libwiredtiger.so.12.0.0+0x471564)
#5 __thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:32:9 (libwiredtiger.so.12.0.0+0x76575c)
Location is heap block of size 912 at 0xffffa0303c00 allocated by main thread:
#0 calloc /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:686:5 (t+0x6c2e4)
#1 __wt_calloc /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_common/os_alloc.c:68:14 (libwiredtiger.so.12.0.0+0x5380a0)
#2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:343:5 (libwiredtiger.so.12.0.0+0x468530)
#3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
#4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
#5 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
#6 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
#7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
Mutex M0 (0xffffa11008f0) created at:
#0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
#1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x31d590)
#2 __wti_connection_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_handle.c:50:5 (libwiredtiger.so.12.0.0+0x31c330)
#3 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3120:5 (libwiredtiger.so.12.0.0+0x2df604)
#4 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
#5 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
#6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
Mutex M1 (0xffffa0303d80) created at:
#0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
#1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x468b78)
#2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:359:5 (libwiredtiger.so.12.0.0+0x4686bc)
#3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
#4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
#5 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
#6 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
#7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
Thread T2 'eviction-ser 1' (tid=7712, running) created by main thread at:
#0 pthread_create /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (t+0x6e014)
#1 __wt_thread_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_posix/os_thread.c:60:5 (libwiredtiger.so.12.0.0+0x553718)
#2 __thread_group_resize /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:209:9 (libwiredtiger.so.12.0.0+0x76341c)
#3 __wt_thread_group_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:295:5 (libwiredtiger.so.12.0.0+0x763d1c)
#4 __wt_evict_threads_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:600:5 (libwiredtiger.so.12.0.0+0x471290)
#5 __wti_connection_workers /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:307:5 (libwiredtiger.so.12.0.0+0x340944)
#6 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3531:5 (libwiredtiger.so.12.0.0+0x2e33ec)
#7 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
#8 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
#9 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
format-stress-test-disagg-leader-tsan task_log
Logs:
SUMMARY: ThreadSanitizer: data race /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 in __wt_atomic_store_double_relaxed
format-stress-test-disagg-leader-tsan task_log
Logs:
WARNING: ThreadSanitizer: data race (pid=7709)
Read of size 8 at 0xffffa0303118 by thread T16 (mutexes: write M0):
#0 __evict_update_work /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:687:30 (libwiredtiger.so.12.0.0+0x47ae2c)
#1 __evict_pass /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:898:9 (libwiredtiger.so.12.0.0+0x479920)
#2 __evict_server /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:463:5 (libwiredtiger.so.12.0.0+0x478c44)
#3 __evict_thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:331:15 (libwiredtiger.so.12.0.0+0x471564)
#4 __thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:32:9 (libwiredtiger.so.12.0.0+0x76575c)
Previous atomic write of size 8 at 0xffffa0303118 by main thread (mutexes: write M1, write M2):
#0 __wt_atomic_store_double_relaxed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 (libwiredtiger.so.12.0.0+0x2bada4)
#1 __checkpoint_update_evict_triggers_end /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:599:17 (libwiredtiger.so.12.0.0+0x2ba35c)
#2 __checkpoint_db_internal /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1739:5 (libwiredtiger.so.12.0.0+0x2b47e8)
#3 __checkpoint_db_wrapper /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1880:11 (libwiredtiger.so.12.0.0+0x2ac834)
#4 __wt_checkpoint_db /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1959:9 (libwiredtiger.so.12.0.0+0x2abf20)
#5 __session_checkpoint /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2443:11 (libwiredtiger.so.12.0.0+0x67c5fc)
#6 wts_load /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/bulk.c:281:9 (t+0x100864)
#7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:370:9 (t+0x134848)
Location is heap block of size 912 at 0xffffa0303000 allocated by main thread:
#0 calloc /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:686:5 (t+0x6c2e4)
#1 __wt_calloc /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_common/os_alloc.c:68:14 (libwiredtiger.so.12.0.0+0x5380a0)
#2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:343:5 (libwiredtiger.so.12.0.0+0x468530)
#3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
#4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
#5 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
#6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
Mutex M0 (0xffffa0303180) created at:
#0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
#1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x468b78)
#2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:359:5 (libwiredtiger.so.12.0.0+0x4686bc)
#3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
#4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
#5 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
#6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
Mutex M1 (0xffffa39a6870) created at:
#0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
#1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x68e9c8)
#2 __open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2602:5 (libwiredtiger.so.12.0.0+0x660804)
#3 __wt_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2727:5 (libwiredtiger.so.12.0.0+0x660094)
#4 __conn_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:1399:5 (libwiredtiger.so.12.0.0+0x2eaed8)
#5 wt_wrap_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/format_util.c:380:5 (t+0x116704)
#6 wts_load /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/bulk.c:278:9 (t+0x1007c8)
#7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:370:9 (t+0x134848)
Mutex M2 (0xffffa11008f0) created at:
#0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
#1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x31d590)
#2 __wti_connection_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_handle.c:50:5 (libwiredtiger.so.12.0.0+0x31c330)
#3 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3120:5 (libwiredtiger.so.12.0.0+0x2df604)
#4 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
#5 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
Thread T16 'eviction-ser 1' (tid=7728, running) created by main thread at:
#0 pthread_create /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (t+0x6e014)
#1 __wt_thread_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_posix/os_thread.c:60:5 (libwiredtiger.so.12.0.0+0x553718)
#2 __thread_group_resize /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:209:9 (libwiredtiger.so.12.0.0+0x76341c)
#3 __wt_thread_group_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:295:5 (libwiredtiger.so.12.0.0+0x763d1c)
#4 __wt_evict_threads_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:600:5 (libwiredtiger.so.12.0.0+0x471290)
#5 __wti_connection_workers /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:307:5 (libwiredtiger.so.12.0.0+0x340944)
#6 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3531:5 (libwiredtiger.so.12.0.0+0x2e33ec)
#7 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
#8 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
format-stress-test-disagg-leader-tsan task_log
Logs:
SUMMARY: ThreadSanitizer: data race /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:687:30 in __evict_update_work
format-stress-test-disagg-leader-tsan task_log
Logs:
#0 0x0000ffffa52bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffa52bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffa52763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffa5262264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000aaaab1a46244 [PAC] in ___interceptor_abort () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1926 #4 0x0000aaaab1a242dc in Abort () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163 #5 0x0000aaaab1a221c4 in __sanitizer::Die() () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58 #6 0x0000aaaab1a8e320 in finalize () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2871 #7 0x0000ffffa5278b58 in __run_exit_handlers () from /lib64/libc.so.6 #8 0x0000ffffa5278c44 [PAC] in exit () from /lib64/libc.so.6 #9 0x0000ffffa526255c [PAC] in __libc_start_call_main () from /lib64/libc.so.6 #10 0x0000ffffa526263c [PAC] in __libc_start_main_impl () from /lib64/libc.so.6 #11 0x0000aaaab1a0d5b0 [PAC] in _start ()
Repro Artifacts:
- is caused by
-
WT-16480 Dynamically tune eviction thresholds during checkpoint
-
- Closed
-