failed: data race on eviction_updates_trigger on amazon2023-arm64-tsan [wiredtiger @ 57b822fe]

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Won't Fix
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Concurrency, Test Format

      format-stress-test-disagg-leader-tsan on amazon2023-arm64-tsan

      Host: i-03328233d2af4985a
      Project: wiredtiger
      Commit: 57b822fe
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-tsan task_log

      Logs:

      WARNING: ThreadSanitizer: data race (pid=7709)
        Atomic write of size 8 at 0xffffa0303d18 by main thread (mutexes: write M0):
          #0 __wt_atomic_store_double_relaxed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 (libwiredtiger.so.12.0.0+0x2bada4)
          #1 __checkpoint_update_evict_triggers_end /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:599:17 (libwiredtiger.so.12.0.0+0x2ba35c)
          #2 __checkpoint_db_internal /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1739:5 (libwiredtiger.so.12.0.0+0x2b47e8)
          #3 __checkpoint_db_wrapper /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1880:11 (libwiredtiger.so.12.0.0+0x2ac834)
          #4 __wt_checkpoint_db /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1959:9 (libwiredtiger.so.12.0.0+0x2abf20)
          #5 __wt_txn_global_shutdown /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/txn/txn.c:2559:17 (libwiredtiger.so.12.0.0+0x786a18)
          #6 __conn_close /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:1261:5 (libwiredtiger.so.12.0.0+0x2e5c50)
          #7 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:763:9 (t+0x1419a0)
          #8 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
        Previous read of size 8 at 0xffffa0303d18 by thread T2 (mutexes: write M1):
          #0 __wti_evict_updates_needed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/../evict/evict_inline.h:532:55 (libwiredtiger.so.12.0.0+0x4782b0)
          #1 __evict_update_work /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:765:9 (libwiredtiger.so.12.0.0+0x47b578)
          #2 __evict_pass /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:898:9 (libwiredtiger.so.12.0.0+0x479920)
          #3 __evict_server /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:463:5 (libwiredtiger.so.12.0.0+0x478c44)
          #4 __evict_thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:331:15 (libwiredtiger.so.12.0.0+0x471564)
          #5 __thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:32:9 (libwiredtiger.so.12.0.0+0x76575c)
        Location is heap block of size 912 at 0xffffa0303c00 allocated by main thread:
          #0 calloc /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:686:5 (t+0x6c2e4)
          #1 __wt_calloc /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_common/os_alloc.c:68:14 (libwiredtiger.so.12.0.0+0x5380a0)
          #2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:343:5 (libwiredtiger.so.12.0.0+0x468530)
          #3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
          #4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
          #5 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
          #6 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
          #7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
        Mutex M0 (0xffffa11008f0) created at:
          #0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
          #1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x31d590)
          #2 __wti_connection_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_handle.c:50:5 (libwiredtiger.so.12.0.0+0x31c330)
          #3 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3120:5 (libwiredtiger.so.12.0.0+0x2df604)
          #4 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
          #5 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
          #6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
        Mutex M1 (0xffffa0303d80) created at:
          #0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
          #1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x468b78)
          #2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:359:5 (libwiredtiger.so.12.0.0+0x4686bc)
          #3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
          #4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
          #5 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
          #6 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
          #7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
        Thread T2 'eviction-ser 1' (tid=7712, running) created by main thread at:
          #0 pthread_create /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (t+0x6e014)
          #1 __wt_thread_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_posix/os_thread.c:60:5 (libwiredtiger.so.12.0.0+0x553718)
          #2 __thread_group_resize /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:209:9 (libwiredtiger.so.12.0.0+0x76341c)
          #3 __wt_thread_group_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:295:5 (libwiredtiger.so.12.0.0+0x763d1c)
          #4 __wt_evict_threads_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:600:5 (libwiredtiger.so.12.0.0+0x471290)
          #5 __wti_connection_workers /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:307:5 (libwiredtiger.so.12.0.0+0x340944)
          #6 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3531:5 (libwiredtiger.so.12.0.0+0x2e33ec)
          #7 create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:602:5 (t+0x13c108)
          #8 wts_create_database /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:753:5 (t+0x1418a0)
          #9 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:357:9 (t+0x13473c)
      

      logs

      format-stress-test-disagg-leader-tsan task_log

      Logs:

      SUMMARY: ThreadSanitizer: data race /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 in __wt_atomic_store_double_relaxed
      

      logs

      format-stress-test-disagg-leader-tsan task_log

      Logs:

      WARNING: ThreadSanitizer: data race (pid=7709)
        Read of size 8 at 0xffffa0303118 by thread T16 (mutexes: write M0):
          #0 __evict_update_work /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:687:30 (libwiredtiger.so.12.0.0+0x47ae2c)
          #1 __evict_pass /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:898:9 (libwiredtiger.so.12.0.0+0x479920)
          #2 __evict_server /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:463:5 (libwiredtiger.so.12.0.0+0x478c44)
          #3 __evict_thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:331:15 (libwiredtiger.so.12.0.0+0x471564)
          #4 __thread_run /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:32:9 (libwiredtiger.so.12.0.0+0x76575c)
        Previous atomic write of size 8 at 0xffffa0303118 by main thread (mutexes: write M1, write M2):
          #0 __wt_atomic_store_double_relaxed /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/gcc.h:426:5 (libwiredtiger.so.12.0.0+0x2bada4)
          #1 __checkpoint_update_evict_triggers_end /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:599:17 (libwiredtiger.so.12.0.0+0x2ba35c)
          #2 __checkpoint_db_internal /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1739:5 (libwiredtiger.so.12.0.0+0x2b47e8)
          #3 __checkpoint_db_wrapper /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1880:11 (libwiredtiger.so.12.0.0+0x2ac834)
          #4 __wt_checkpoint_db /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/checkpoint/checkpoint_txn.c:1959:9 (libwiredtiger.so.12.0.0+0x2abf20)
          #5 __session_checkpoint /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2443:11 (libwiredtiger.so.12.0.0+0x67c5fc)
          #6 wts_load /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/bulk.c:281:9 (t+0x100864)
          #7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:370:9 (t+0x134848)
        Location is heap block of size 912 at 0xffffa0303000 allocated by main thread:
          #0 calloc /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:686:5 (t+0x6c2e4)
          #1 __wt_calloc /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_common/os_alloc.c:68:14 (libwiredtiger.so.12.0.0+0x5380a0)
          #2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:343:5 (libwiredtiger.so.12.0.0+0x468530)
          #3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
          #4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
          #5 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
          #6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
        Mutex M0 (0xffffa0303180) created at:
          #0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
          #1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x468b78)
          #2 __wt_evict_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_conn.c:359:5 (libwiredtiger.so.12.0.0+0x4686bc)
          #3 __wti_connection_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:53:5 (libwiredtiger.so.12.0.0+0x33d4ac)
          #4 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3413:5 (libwiredtiger.so.12.0.0+0x2e24ec)
          #5 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
          #6 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
        Mutex M1 (0xffffa39a6870) created at:
          #0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
          #1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x68e9c8)
          #2 __open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2602:5 (libwiredtiger.so.12.0.0+0x660804)
          #3 __wt_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/session/session_api.c:2727:5 (libwiredtiger.so.12.0.0+0x660094)
          #4 __conn_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:1399:5 (libwiredtiger.so.12.0.0+0x2eaed8)
          #5 wt_wrap_open_session /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/format_util.c:380:5 (t+0x116704)
          #6 wts_load /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/bulk.c:278:9 (t+0x1007c8)
          #7 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:370:9 (t+0x134848)
        Mutex M2 (0xffffa11008f0) created at:
          #0 pthread_mutex_init /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1327:3 (t+0x6fef4)
          #1 __wt_spin_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/include/mutex_inline.h:142:5 (libwiredtiger.so.12.0.0+0x31d590)
          #2 __wti_connection_init /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_handle.c:50:5 (libwiredtiger.so.12.0.0+0x31c330)
          #3 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3120:5 (libwiredtiger.so.12.0.0+0x2df604)
          #4 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
          #5 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
        Thread T16 'eviction-ser 1' (tid=7728, running) created by main thread at:
          #0 pthread_create /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1023:3 (t+0x6e014)
          #1 __wt_thread_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/os_posix/os_thread.c:60:5 (libwiredtiger.so.12.0.0+0x553718)
          #2 __thread_group_resize /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:209:9 (libwiredtiger.so.12.0.0+0x76341c)
          #3 __wt_thread_group_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/support/thread_group.c:295:5 (libwiredtiger.so.12.0.0+0x763d1c)
          #4 __wt_evict_threads_create /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:600:5 (libwiredtiger.so.12.0.0+0x471290)
          #5 __wti_connection_workers /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_open.c:307:5 (libwiredtiger.so.12.0.0+0x340944)
          #6 wiredtiger_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/conn/conn_api.c:3531:5 (libwiredtiger.so.12.0.0+0x2e33ec)
          #7 wts_open /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/wts.c:842:9 (t+0x143884)
          #8 main /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/test/format/t.c:358:9 (t+0x134754)
      

      logs

      format-stress-test-disagg-leader-tsan task_log

      Logs:

      SUMMARY: ThreadSanitizer: data race /data/mci/666175727707a74b340745fed1014fd6/wiredtiger/src/evict/evict_lru.c:687:30 in __evict_update_work
      

      logs

      format-stress-test-disagg-leader-tsan task_log

      Logs:

      #0  0x0000ffffa52bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffffa52bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffffa52763a0 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffffa5262264 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000aaaab1a46244 [PAC] in ___interceptor_abort () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1926
      #4  0x0000aaaab1a242dc in Abort () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163
      #5  0x0000aaaab1a221c4 in __sanitizer::Die() () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58
      #6  0x0000aaaab1a8e320 in finalize () at /data/mci/f49c90f0be13703fc74cd438c017c2d0/toolchain-builder/tmp/build-llvm-v5.sh-WmZ/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2871
      #7  0x0000ffffa5278b58 in __run_exit_handlers () from /lib64/libc.so.6
      #8  0x0000ffffa5278c44 [PAC] in exit () from /lib64/libc.so.6
      #9  0x0000ffffa526255c [PAC] in __libc_start_call_main () from /lib64/libc.so.6
      #10 0x0000ffffa526263c [PAC] in __libc_start_main_impl () from /lib64/libc.so.6
      #11 0x0000aaaab1a0d5b0 [PAC] in _start ()
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: