failed: format-stress-test-disagg-leader-data-validation-1 on amazon2023-arm64-asan [wiredtiger @ 8fa1e5db]

XMLWordPrintableJSON

      format-stress-test-disagg-leader-data-validation-1 on amazon2023-arm64-asan

      Host: i-0e7e70e797a641a2f
      Project: wiredtiger
      Commit: 8fa1e5db
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      ==33956==ERROR: AddressSanitizer: heap-use-after-free on address 0x503000edb9b0 at pc 0xffff9ad05afc bp 0xffff28aeb1e0 sp 0xffff28aea9d0
      READ of size 20 at 0x503000edb9b0 thread T56
          #0 0xffff9ad05af8 in __asan_memcpy /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63:3
          #1 0xffff9a1b23f4 in __wti_rec_kv_copy /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/reconcile_inline.h:336:9
          #2 0xffff9a1bb834 in __rec_pack_delta_row_int /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_row.c:296:9
          #3 0xffff9a1b9da4 in __wti_rec_row_int /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_row.c:814:13
          #4 0xffff9a21e688 in __reconcile /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:362:9
          #5 0xffff9a219d78 in __wt_reconcile /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:194:11
          #6 0xffff99b10ea4 in __wt_sync_file /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_sync.c:473:17
          #7 0xffff99c48694 in __checkpoint_tree /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3148:9
          #8 0xffff99c698d4 in __checkpoint_tree_helper /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3337:11
          #9 0xffff99c5aa08 in __checkpoint_apply_to_dhandles /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:372:9
          #10 0xffff99c59404 in __checkpoint_selected_dhandles /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3419:5
          #11 0xffff99c4e8a8 in __checkpoint_db_internal /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:1862:5
          #12 0xffff99c45a5c in __checkpoint_db_wrapper /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:2125:11
          #13 0xffff99c44b14 in __wt_checkpoint_db /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:2204:9
          #14 0xffff9a326700 in __session_checkpoint /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/session/session_api.c:2508:11
          #15 0xaaaabeccd194 in checkpoint /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/checkpoint.c:148:15
          #16 0xffff9ad056dc in asan_thread_start(void*) /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
          #17 0xffff9aac2830 in start_thread (/lib64/libc.so.6+0x90830) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
          #18 0xffff9aa66e58 in thread_start (/lib64/libc.so.6+0x34e58) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
      0x503000edb9b0 is located 0 bytes inside of 20-byte region [0x503000edb9b0,0x503000edb9c4)
      freed by thread T40 here:
          #0 0xffff9ad07c40 in free /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_malloc_linux.cpp:52:3
          #1 0xffff9a0e973c in __wt_free_int /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_common/os_alloc.c:274:5
          #2 0xffff999b4c70 in __free_page_modify /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_discard.c:230:9
          #3 0xffff999b1dc0 in __wt_page_out /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_discard.c:142:9
          #4 0xffff999b06fc in __wt_ref_out /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_discard.c:43:5
          #5 0xffff99abdc54 in __wt_split_rewrite /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_split.c:2588:5
          #6 0xffff99fa2b08 in __evict_page_dirty_update /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/evict/evict_page.c:808:19
          #7 0xffff99f9ba48 in __wt_evict /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/evict/evict_page.c:561:9
          #8 0xffff99a53ac8 in __wt_page_release_evict /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_read.c:141:11
          #9 0xffff9994f768 in __wt_page_release /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/include/btree_inline.h:2637:9
          #10 0xffff9992e1e8 in __cursor_reset /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/include/cursor_inline.h:309:15
          #11 0xffff9992fc0c in __wt_btcur_search /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_cursor.c:956:9
          #12 0xffff99dffbf4 in __curfile_search /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/cursor/cur_file.c:325:5
          #13 0xffff99eacef4 in __clayered_lookup_constituent /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/cursor/cur_layered.c:2258:16
          #14 0xffff99eac6c8 in __clayered_lookup /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/cursor/cur_layered.c:2357:9
          #15 0xffff99e79eb0 in __clayered_search /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/cursor/cur_layered.c:2394:5
          #16 0xaaaabed198b0 in read_op /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/format_inline.h:51:15
          #17 0xaaaabed05520 in read_row_worker /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:1622:15
          #18 0xaaaabed0d3a4 in read_row /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:1803:13
          #19 0xaaaabed0a798 in table_op /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:971:15
          #20 0xaaaabed02428 in ops /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:1433:19
          #21 0xffff9ad056dc in asan_thread_start(void*) /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
          #22 0x1fffff9aa66e58  (<unknown module>)
      previously allocated by thread T19 here:
          #0 0xffff9ad07eb0 in malloc /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_malloc_linux.cpp:68:3
          #1 0xffff9a0e7d0c in __wt_malloc /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_common/os_alloc.c:99:14
          #2 0xffff9a0e8eb0 in __wt_memdup /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_common/os_alloc.c:210:5
          #3 0xffff9a22fdac in __rec_split_write /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:2733:5
          #4 0xffff9a226c2c in __wti_rec_split_finish /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:1806:13
          #5 0xffff9a1c6e0c in __wti_rec_row_leaf /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_row.c:1416:11
          #6 0xffff9a21ece0 in __reconcile /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:374:9
          #7 0xffff9a219d78 in __wt_reconcile /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:194:11
          #8 0xffff99c27e44 in __checkpoint_parallel_thread_run /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_parallel.c:235:9
          #9 0xffff9a493394 in __thread_run /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/support/thread_group.c:32:9
          #10 0xffff9ad056dc in asan_thread_start(void*) /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
          #11 0x11ffff9aa66e58  (<unknown module>)
      Thread T56 created by T0 here:
          #0 0xffff9acecb74 in pthread_create /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
          #1 0xffff9a112a60 in __wt_thread_create /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_posix/os_thread.c:71:5
          #2 0xaaaabecfccb8 in operations /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:403:9
          #3 0xaaaabed2868c in main /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/t.c:401:13
          #4 0xffff9aa67a74 in __libc_start_call_main (/lib64/libc.so.6+0x35a74) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
          #5 0x5dffff9aa67b58  (<unknown module>)
          #6 0x42aaaabecc2a2c  (<unknown module>)
      Thread T40 created by T0 here:
          #0 0xffff9acecb74 in pthread_create /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
          #1 0xffff9a112a60 in __wt_thread_create /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_posix/os_thread.c:71:5
          #2 0xaaaabecfc3c4 in operations /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/ops.c:379:9
          #3 0xaaaabed2868c in main /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/t.c:401:13
          #4 0xffff9aa67a74 in __libc_start_call_main (/lib64/libc.so.6+0x35a74) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
          #5 0x5dffff9aa67b58  (<unknown module>)
          #6 0x42aaaabecc2a2c  (<unknown module>)
      Thread T19 created by T0 here:
          #0 0xffff9acecb74 in pthread_create /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
          #1 0xffff9a112a60 in __wt_thread_create /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/os_posix/os_thread.c:71:5
          #2 0xffff9a48f60c in __thread_group_resize /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/support/thread_group.c:206:9
          #3 0xffff9a490274 in __wt_thread_group_create /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/support/thread_group.c:290:5
          #4 0xffff99c27414 in __wt_checkpoint_parallel_thread_create /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_parallel.c:329:5
          #5 0xffff99d61aac in __wti_connection_workers /data/mci/
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      SUMMARY: AddressSanitizer: heap-use-after-free /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/reconcile_inline.h:336:9 in __wti_rec_kv_copy
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      Shadow bytes around the buggy address:
        0x503000edb700: fa fa fd fd fd fd fa fa fd fd fd fd fa fa 00 00
        0x503000edb780: 03 fa fa fa 00 00 00 fa fa fa 00 00 03 fa fa fa
        0x503000edb800: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fa
        0x503000edb880: fa fa fa fa fa fa fa fa 00 00 03 fa fa fa fd fd
        0x503000edb900: fd fd fa fa fa fa fa fa fa fa fd fd fd fa fa fa
      =>0x503000edb980: 00 00 02 fa fa fa[fd]fd fd fa fa fa fd fd fd fd
        0x503000edba00: fa fa 00 00 02 fa fa fa 00 00 02 fa fa fa 00 00
        0x503000edba80: 02 fa fa fa fd fd fd fd fa fa fa fa fa fa fa fa
        0x503000edbb00: 00 00 03 fa fa fa fa fa fa fa fa fa 00 00 02 fa
        0x503000edbb80: fa fa 00 00 02 fa fa fa 00 00 02 fa fa fa 00 00
        0x503000edbc00: 02 fa fa fa 00 00 00 01 fa fa fd fd fd fd fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:       fa
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      #0  0x0000ffff9aac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff9aac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff9aa7b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff9aa62224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9ac5127c [PAC] in Abort () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163
      #4  0x0000ffff9ac4f164 in __sanitizer::Die() () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58
      #5  0x0000ffff9ad0cac0 in ~ScopedInErrorReport () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:193
      #6  0x0000ffff9ad0fe00 in ReportGenericError () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:498
      #7  0x0000ffff9ad05b24 in __asan_memcpy () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63
      #8  0x0000ffff9a1b23f8 in __wti_rec_kv_copy (session=0xffff98e417c0, p=0x532008534956 '\253' <repeats 200 times>..., kv=0x51e001a765f8) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/reconcile_inline.h:336
      #9  0x0000ffff9a1bb838 in __rec_pack_delta_row_int (session=0xffff98e417c0, r=0x51e001a76080, key=0x51e001a76558, value=0x51e001a765f8, ta=0xffff261d9240) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_row.c:296
      #10 0x0000ffff9a1b9da8 in __wti_rec_row_int (session=0xffff98e417c0, r=0x51e001a76080, page=0x5120009a2c40) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_row.c:814
      #11 0x0000ffff9a21e68c in __reconcile (session=0xffff98e417c0, ref=0x50e00038ed60, salvage=0x0, flags=132, page_lockedp=0xffff2610b020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:362
      #12 0x0000ffff9a219d7c in __wt_reconcile (session=0xffff98e417c0, ref=0x50e00038ed60, salvage=0x0, flags=132) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/reconcile/rec_write.c:194
      #13 0x0000ffff99b10ea8 in __wt_sync_file (session=0xffff98e417c0, syncop=WT_SYNC_CHECKPOINT) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/btree/bt_sync.c:473
      #14 0x0000ffff99c48698 in __checkpoint_tree (session=0xffff98e417c0, is_checkpoint=true, cfg=0xffff2600e020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3148
      #15 0x0000ffff99c698d8 in __checkpoint_tree_helper (session=0xffff98e417c0, cfg=0xffff2600e020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3337
      #16 0x0000ffff99c5aa0c in __checkpoint_apply_to_dhandles (session=0xffff98e417c0, cfg=0xffff2600e020, op=0xffff99c694b0 <__checkpoint_tree_helper>) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:372
      #17 0x0000ffff99c59408 in __checkpoint_selected_dhandles (session=0xffff98e417c0, cfg=0xffff2600e020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:3419
      #18 0x0000ffff99c4e8ac in __checkpoint_db_internal (session=0xffff98e417c0, cfg=0xffff2600e020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:1862
      #19 0x0000ffff99c45a60 in __checkpoint_db_wrapper (session=0xffff98e417c0, cfg=0xffff2600e020) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:2125
      #20 0x0000ffff99c44b18 in __wt_checkpoint_db (session=0xffff98e417c0, cfg=0xffff2600e020, waiting=true) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/checkpoint/checkpoint_txn.c:2204
      #21 0x0000ffff9a326704 in __session_checkpoint (wt_session=0xffff98e417c0, config=0x0) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/src/session/session_api.c:2508
      #22 0x0000aaaabeccd198 in checkpoint (arg=0x0) at /data/mci/e96c01c65bb9ed38beedefcbf43ce355/wiredtiger/test/format/checkpoint.c:148
      #23 0x0000ffff9ad056e0 in asan_thread_start () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239
      #24 0x0000ffff9aac2834 in start_thread () from /lib64/libc.so.6
      #25 0x0000ffff9aa66e5c [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            Chenhao Qu
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: