-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines, Storage Engines - Foundations
-
None
-
None
-
1
format-stress-test-disagg-switch-1 on amazon2023-disagg-asan-stress
Host: i-05c317e4a6d7ec0a1
Project: wiredtiger
Commit: 29aa0a01
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-switch-1 task_log
Logs:
==27699==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5030008e4bd0 at pc 0xffffb169911c bp 0xffff738b1230 sp 0xffff738b1228
READ of size 8 at 0x5030008e4bd0 thread T87
#0 0xffffb1699118 in __layered_drain_worker_run /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:2070:5
#1 0xffffb1d5c8b4 in __thread_run /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/support/thread_group.c:32:9
#2 0xffffb250579c in asan_thread_start(void*) /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
#3 0xffffb0ebdb74 in start_thread (/lib64/libc.so.6+0x8bb74) (BuildId: 7f46fcd5f30f2f31235144f8192c97c839732436)
#4 0xffffb0f2acd8 in thread_start (/lib64/libc.so.6+0xf8cd8) (BuildId: 7f46fcd5f30f2f31235144f8192c97c839732436)
Address 0x5030008e4bd0 is a wild pointer inside of access range of size 0x000000000008.
format-stress-test-disagg-switch-1 task_log
Logs:
SUMMARY: AddressSanitizer: heap-buffer-overflow /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:2070:5 in __layered_drain_worker_run
format-stress-test-disagg-switch-1 task_log
Logs:
Shadow bytes around the buggy address:
0x5030008e4900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4a80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4b00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x5030008e4b80: fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa fa fa
0x5030008e4c00: fa fa fa fa fd fd fd fa fa fa fa fa fa fa fa fa
0x5030008e4c80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4d00: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
0x5030008e4d80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x5030008e4e00: 00 00 00 fa fa fa fa fa fa fa fa fa 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Thread T87 created by T0 here:
#0 0xffffb24ecc34 in pthread_create /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
#1 0xffffb1a1de04 in __wt_thread_create /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/os_posix/os_thread.c:60:5
#2 0xffffb1d58e14 in __thread_group_resize /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/support/thread_group.c:209:9
#3 0xffffb1d59ad8 in __wt_thread_group_create /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/support/thread_group.c:295:5
#4 0xffffb1697850 in __layered_drain_ingest_tables /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:2158:9
#5 0xffffb1688380 in __disagg_step_up /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:1306:5
#6 0xffffb1684830 in __wti_disagg_conn_config /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:1406:9
#7 0xffffb16b8f44 in __wti_conn_reconfig /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_reconfig.c:450:13
#8 0xffffb1622124 in __conn_reconfigure /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_api.c:1354:11
#9 0xaaaadab15658 in disagg_switch_roles /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/test/format/format_disagg.c:203:9
#10 0xaaaadab4ed0c in main /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/test/format/t.c:407:13
#11 0xffffb0e62554 in __libc_start_call_main (/lib64/libc.so.6+0x30554) (BuildId: 7f46fcd5f30f2f31235144f8192c97c839732436)
#12 0x44ffffb0e62638 (<unknown module>)
#13 0x11aaaadaaeff2c (<unknown module>)
format-stress-test-disagg-switch-1 task_log
Logs:
#0 0x0000ffffb0ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffb0ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffb0e763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffb0e62264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffffb245133c [PAC] in Abort () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163 #4 0x0000ffffb244f224 in __sanitizer::Die() () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58 #5 0x0000ffffb250cb80 in ~ScopedInErrorReport () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:193 #6 0x0000ffffb250fec0 in ReportGenericError () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:498 #7 0x0000ffffb2510dd0 in __asan_report_load8 () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_rtl.cpp:131 #8 0x0000ffffb169911c in __layered_drain_worker_run (session=0xffffb0518c48, ctx=0x507003c3b110) at /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/conn/conn_layered.c:2070 #9 0x0000ffffb1d5c8b8 in __thread_run (arg=0x507003c3b110) at /data/mci/e8c5f63b75a31307a6901ef2341a59d1/wiredtiger/src/support/thread_group.c:32 #10 0x0000ffffb25057a0 in asan_thread_start () at /data/mci/01bb46477e468e9b17d7d0a0c518db71/toolchain-builder/tmp/build-llvm-v5.sh-42U/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239 #11 0x0000ffffb0ebdb78 in start_thread () from /lib64/libc.so.6 #12 0x0000ffffb0f2acdc [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts: