[SERVER-68656] Compile our own libstdc++ for MSan Created: 08/Aug/22  Updated: 02/Feb/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Alex Neben Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-67132 Add MSan to SCons Closed
Assigned Teams:
Build
Participants:

 Description   

From SERVER-67132 we learned that we need to compile our own libstdc++ to get meaningful data from msan in scons. Right now it shows errors that are clearly just due to a lack of augmented libstdc++.

[j0] ==28254==WARNING: MemorySanitizer: use-of-uninitialized-value
[j0]     #0 0x7fb90d99116d in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::_M_lower_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >*, std::_Rb_tree_node_base*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:1894:7
[j0]     #1 0x7fb90d99116d in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:1210
[j0]     #2 0x7fb90d99116d in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::lower_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_map.h:1239
[j0]     #3 0x7fb90d99116d in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_map.h:495
[j0]     #4 0x7fb90d98c39f in mongo::MetricTree::_add(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<mongo::ServerStatusMetric, std::default_delete<mongo::ServerStatusMetric> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.cpp:95:17
[j0]     #5 0x7fb90d98b818 in mongo::MetricTree::add(std::unique_ptr<mongo::ServerStatusMetric, std::default_delete<mongo::ServerStatusMetric> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.cpp:76:9
[j0]     #6 0x7fb921e8e874 in mongo::ServerStatusMetricField<mongo::Counter64>& mongo::addMetricToTree<mongo::ServerStatusMetricField<mongo::Counter64> >(std::unique_ptr<mongo::ServerStatusMetricField<mongo::Counter64>, std::default_delete<mongo::ServerStatusMetricField<mongo::Counter64> > >, mongo::MetricTree*) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:138:17
[j0]     #7 0x7fb921e8e543 in mongo::Counter64& mongo::makeServerStatusMetric<mongo::Counter64>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:144:12
[j0]     #8 0x7fb921e7e3b0 in mongo::CounterMetric::CounterMetric(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:151:20
[j0]     #9 0x7fb90ffb879b in mongo::DotsAndDollarsFieldsCounters::DotsAndDollarsFieldsCounters() /home/ubuntu/mongo/src/mongo/db/stats/counters.h:306:26
[j0]     #10 0x7fb90ffad8f4 in __cxx_global_var_init.55 /home/ubuntu/mongo/src/mongo/db/stats/counters.cpp:320:30
[j0]     #11 0x7fb90ffad8f4 in _GLOBAL__sub_I__ /home/ubuntu/.ccache/tmp/counters.stdout.ip-10-122-6-50.31190.2RZy4l.ii
[j0]     #12 0x7fb9278b08d2 in call_init /build/glibc-uZu3wS/glibc-2.27/elf/dl-init.c:72
[j0]     #13 0x7fb9278b08d2 in _dl_init /build/glibc-uZu3wS/glibc-2.27/elf/dl-init.c:119
[j0]     #14 0x7fb9278a10c9  (/lib64/ld-linux-x86-64.so.2+0x10c9)
[j0]
[j0]   Uninitialized value was created by a heap allocation
[j0]     #0 0x558ac3a244c9 in operator new(unsigned long) /data/mci/bf9fe9fe7099cc84c75c9ab85373ed19/toolchain-builder/tmp/build-llvm.sh-nzs/llvm/projects/compiler-rt/lib/msan/msan_new_delete.cc:46:35
[j0]     #1 0x7fb90d99d64c in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::allocate(unsigned long, void const*) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/ext/new_allocator.h:111:27
[j0]     #2 0x7fb90d99d64c in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >&, unsigned long) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/alloc_traits.h:436
[j0]     #3 0x7fb90d99d64c in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::_M_get_node() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:588
[j0]     #4 0x7fb90d99d64c in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:642
[j0]     #5 0x7fb90d99d0d1 in std::_Rb_tree_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::piecewise_construct_t const&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:2421:19
[j0]     #6 0x7fb90d99108b in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_map.h:499:15
[j0]     #7 0x7fb90d98c39f in mongo::MetricTree::_add(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<mongo::ServerStatusMetric, std::default_delete<mongo::ServerStatusMetric> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.cpp:95:17
[j0]     #8 0x7fb90d98b818 in mongo::MetricTree::add(std::unique_ptr<mongo::ServerStatusMetric, std::default_delete<mongo::ServerStatusMetric> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.cpp:76:9
[j0]     #9 0x7fb921e8e874 in mongo::ServerStatusMetricField<mongo::Counter64>& mongo::addMetricToTree<mongo::ServerStatusMetricField<mongo::Counter64> >(std::unique_ptr<mongo::ServerStatusMetricField<mongo::Counter64>, std::default_delete<mongo::ServerStatusMetricField<mongo::Counter64> > >, mongo::MetricTree*) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:138:17
[j0]     #10 0x7fb921e8e543 in mongo::Counter64& mongo::makeServerStatusMetric<mongo::Counter64>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:144:12
[j0]     #11 0x7fb921e7e3b0 in mongo::CounterMetric::CounterMetric(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) /home/ubuntu/mongo/src/mongo/db/commands/server_status_metric.h:151:20
[j0]     #12 0x7fb90ffb864f in mongo::DotsAndDollarsFieldsCounters::DotsAndDollarsFieldsCounters() /home/ubuntu/mongo/src/mongo/db/stats/counters.h:305:26
[j0]     #13 0x7fb90ffad8f4 in __cxx_global_var_init.55 /home/ubuntu/mongo/src/mongo/db/stats/counters.cpp:320:30
[j0]     #14 0x7fb90ffad8f4 in _GLOBAL__sub_I__ /home/ubuntu/.ccache/tmp/counters.stdout.ip-10-122-6-50.31190.2RZy4l.ii
[j0]     #15 0x7fb9278b08d2 in call_init /build/glibc-uZu3wS/glibc-2.27/elf/dl-init.c:72
[j0]     #16 0x7fb9278b08d2 in _dl_init /build/glibc-uZu3wS/glibc-2.27/elf/dl-init.c:119
[j0]
[j0] SUMMARY: MemorySanitizer: use-of-uninitialized-value /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.JnX/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_tree.h:1894:7 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mongo::MetricTree, std::default_delete<mongo::MetricTree> > > > >::_M_lower_bound(std::_Rb_tree_node<std
[j0] Exiting
[j0] Waiting to connect to mongod on port 20000. 



 Comments   
Comment by Andrew Morrow (Inactive) [ 29/Sep/22 ]

alexander.neben@mongodb.com - My understanding from reading the MSAN documents was that it isn't just libstdc++ that you need to build in order to avoid false positives with MSAN, but basically all system libraries that you use. In our case that would include things like openssl, libldap, etc.

From https://clang.llvm.org/docs/MemorySanitizer.html

MemorySanitizer requires that all program code is instrumented. This also includes any libraries that the program depends on, even libc. Failing to achieve this may result in false reports. For the same reason you may need to replace all inline assembly code that writes to memory with a pure C/C++ code.

Full MemorySanitizer instrumentation is very difficult to achieve. To make it easier, MemorySanitizer runtime library includes 70+ interceptors for the most common libc functions. They make it possible to run MemorySanitizer-instrumented programs linked with uninstrumented libc. For example, the authors were able to bootstrap MemorySanitizer-instrumented Clang compiler by linking it with self-built instrumented libc++ (as a replacement for libstdc++).

I've had a longstanding idea about how to approach this, which amounts to using gentoo or another source based linux distribution where we could create a container where the entire userspace had been rebuilt with -fsanitize=memory. We would then build the server project on a different container built without that flag (since we don't want to sanitize the build), and then run it in the sanitized userspace container.

Generated at Thu Feb 08 06:11:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.