[SERVER-77770] Ensure that TSAN works on ARM architecture Created: 02/Jun/23  Updated: 02/Feb/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Trivial - P5
Reporter: Daniel Moody Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-78018 TSAN crashes on symbolization Closed
Assigned Teams:
Build
Operating System: ALL
Steps To Reproduce:

git checkout 528d57bf
python3 ./buildscripts/scons.py --modules= --dbg=on --opt=on --use-libunwind=off --variables-files=./etc/scons/mongodbtoolchain_stable_clang.vars --ninja ICECC=icecc CCACHE=ccache --sanitize=thread --allocator=system NINJA_PREFIX=tsan VARIANT_DIR=tsandebug
ninja -f tsan.ninja install-devcore -j 200
TSAN_OPTIONS="abort_on_error=1:disable_coredump=0:handle_abort=1:halt_on_error=1:report_thread_leaks=0:die_after_fork=0:suppressions=etc/tsan.suppressions:external_symbolizer_path=/opt/mongodbtoolchain/v4/bin/llvm-symbolizer" buildscripts/resmoke.py run --suites=jstestfuzz_sharded --numClientsPerFixture=10 --storageEngineCacheSizeGB=1 ~/temp/out/* > output.txt

Participants:
Linked BF Score: 135

 Description   
  1. Adapt the build system as necessary to ensure that TSAN works on ARM architecture.
  2. Ensure that at-least-one Evergreen variant exercises this configuration (so we can detect if/when it breaks).


 Comments   
Comment by Ryan Scott [ 26/Jun/23 ]

We've only seen one instance of this so far. If/when this occurs more frequently SDP team will reprioritize. 

Comment by Alex Neben [ 19/Jun/23 ]

That did not work . dynamic still has the same problem.

Comment by Alex Neben [ 19/Jun/23 ]

It looks like we are going to ban link model static anyway as part of SERVER-78018. It looks like that might solve this issue. I guess we would need to see if this goes away on dynamic builds.

Comment by Alex Neben [ 06/Jun/23 ]

I played with this one for a while. I tried lots of googling and then in a fit of guessing I set JS_64BIT 0. However, nothing worked.

Comment by Alex Neben [ 06/Jun/23 ]

backtrace

(mongo) ubuntu@ip-10-122-5-130:~/mongo$ TSAN_OPTIONS="allocator_may_return_null=1:report_signal_unsafe=0:abort_on_error=1:disable_coredump=0:handle_abort=1:halt_on_error=1:report_thread_leaks=0:die_after_fork=0:suppressions=etc/tsan.suppressions:external_symbolizer_path=/opt/mongodbtoolchain/v4/bin/llvm-symbolizer" gdb --args ./build/install/bin/mongo --nodb
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/install/bin/mongo...
r
bt
warning: File "/home/ubuntu/mongo/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/ubuntu/mongo/.gdbinit
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
--Type <RET> for more, q to quit, c to continue without paging--r
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/ubuntu/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) bt
No stack.
(gdb) r
Starting program: /home/ubuntu/mongo/build/install/bin/mongo --nodb
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
[New Thread 0xffffeefbf080 (LWP 3726755)]
[New Thread 0xffffee7af080 (LWP 3726756)]
MongoDB shell version v7.1.0-alpha-1144-g528d57b
[New Thread 0xffffedf9f080 (LWP 3726757)]
[New Thread 0xffffed78f080 (LWP 3726758)]
[Thread 0xffffed78f080 (LWP 3726758) exited]
[New Thread 0xffffed78f080 (LWP 3726759)]Thread 6 "js" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xffffed78f080 (LWP 3726759)]
0x0000aaaaad7a3f00 in js::gc::MapAlignedPagesRandom (length=<optimized out>, alignment=1048576) at src/third_party/mozjs/extract/js/src/gc/Memory.cpp:578
578         MOZ_CRASH("Couldn't allocate even after 1000 tries!");
(gdb) bt
#0  0x0000aaaaad7a3f00 in js::gc::MapAlignedPagesRandom (length=<optimized out>, alignment=1048576) at src/third_party/mozjs/extract/js/src/gc/Memory.cpp:578
#1  js::gc::MapAlignedPages (length=<optimized out>, length@entry=1048576, alignment=alignment@entry=1048576) at src/third_party/mozjs/extract/js/src/gc/Memory.cpp:449
#2  0x0000aaaaad73da18 in js::gc::TenuredChunk::allocate (gc=0xffffef80a528) at src/third_party/mozjs/extract/js/src/gc/Allocator.cpp:878
#3  js::gc::GCRuntime::getOrAllocChunk (this=0xffffef80a528, lock=...) at src/third_party/mozjs/extract/js/src/gc/Allocator.cpp:809
#4  0x0000aaaaad7a5830 in js::Nursery::allocateNextChunk (this=0xffffef80c6b0, chunkno=0, lock=...) at src/third_party/mozjs/extract/js/src/gc/Nursery.cpp:1542
#5  js::Nursery::initFirstChunk (this=this@entry=0xffffef80c6b0, lock=...) at src/third_party/mozjs/extract/js/src/gc/Nursery.cpp:290
#6  0x0000aaaaad7a5658 in js::Nursery::init (this=0xffffef80c6b0, lock=...) at src/third_party/mozjs/extract/js/src/gc/Nursery.cpp:254
#7  0x0000aaaaad7449cc in js::gc::GCRuntime::init (this=0xffffef80a528, maxbytes=4294967295) at src/third_party/mozjs/extract/js/src/gc/GC.cpp:1565
#8  0x0000aaaaad0f5c04 in JSRuntime::init (this=0xffffef80a000, cx=<optimized out>, maxbytes=4294967295) at src/third_party/mozjs/extract/js/src/vm/Runtime.cpp:211
#9  0x0000aaaaacfe72c8 in js::NewContext (maxBytes=maxBytes@entry=4294967295, parentRuntime=parentRuntime@entry=0x0) at src/third_party/mozjs/extract/js/src/vm/JSContext.cpp:200
#10 0x0000aaaaad261570 in JS_NewContext (maxbytes=4294967295, parentRuntime=0x0) at src/third_party/mozjs/extract/js/src/jsapi.cpp:405
#11 0x0000aaaaacc9121c in mongo::mozjs::MozJSImplScope::MozRuntime::MozRuntime (this=this@entry=0xfffff001b0d0, engine=engine@entry=0xfffff57b0a00, jsHeapLimitMB=...)
    at src/mongo/scripting/mozjs/implscope.cpp:338
#12 0x0000aaaaacc91a74 in mongo::mozjs::MozJSImplScope::MozJSImplScope (this=0xfffff001b000, engine=0xfffff57b0a00, jsHeapLimitMB=...) at src/mongo/scripting/mozjs/implscope.cpp:436
#13 0x0000aaaaacd021dc in mongo::mozjs::MozJSProxyScope::implThread (proxy=0xfffff2a00f00) at src/mongo/scripting/mozjs/proxyscope.cpp:382
#14 0x0000aaaaacd0a4cc in std::__invoke_impl<void, void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*> (__f=<optimized out>, __args=@0xfffff5533758: 0xfffff2a00f00)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:61
#15 std::__invoke<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*> (__fn=<optimized out>, __args=@0xfffff5533758: 0xfffff2a00f00)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:96
--Type <RET> for more, q to quit, c to continue without paging--
#16 std::__apply_impl<void (*)(mongo::mozjs::MozJSProxyScope*), std::tuple<mongo::mozjs::MozJSProxyScope*>, 0ul> (__f=<optimized out>, __t=...)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/tuple:1858
#17 std::apply<void (*)(mongo::mozjs::MozJSProxyScope*), std::tuple<mongo::mozjs::MozJSProxyScope*> > (__f=<optimized out>, __t=...)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/tuple:1869
#18 mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}::operator()() (this=this@entry=0xfffff5533748) at src/mongo/stdx/thread.h:192
#19 0x0000aaaaacd0a454 in std::__invoke_impl<void, mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}>(std::__invoke_other, mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}&&) (__f=...)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:61
#20 std::__invoke<mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}>(mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}&&) (__fn=...)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:96
#21 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0xfffff5533748)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_thread.h:253
#22 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}> >::operator()() (this=0xfffff5533748)
--Type <RET> for more, q to quit, c to continue without paging--
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_thread.h:260
#23 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*, 0>(void (*)(mongo::mozjs::MozJSProxyScope*), mongo::mozjs::MozJSProxyScope*&&)::{lambda()#1}> > >::_M_run() (this=0xfffff5533740)
    at /opt/mongodbtoolchain/revisions/11316f1e7b36f08dcdd2ad0640af18f9287876f4/stow/gcc-v4.XAW/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_thread.h:211
#24 0x0000aaaaaee1d96c in execute_native_thread_routine ()
#25 0x0000aaaaac5cc4bc in __tsan_thread_start_func ()
    at /data/mci/f98384436d832a2873cf850ba75a0da0/toolchain-builder/tmp/build-llvm-v4.sh-EML/llvm-project-llvmorg/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:955
#26 0x0000fffff781d5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
#27 0x0000fffff7885d1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79 

Generated at Thu Feb 08 06:36:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.