[SERVER-24001] Add a RocksDB+ASan build variant to Evergreen Created: 01/May/16  Updated: 26/Sep/18  Resolved: 26/Sep/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Minor - P4
Reporter: Max Hirschhorn Assignee: DO NOT USE - Backlog - Test Infrastructure Group (TIG)
Resolution: Won't Fix Votes: 0
Labels: tig-evgconfig
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File compile_output.log     File rocks_asan_evg.diff    
Issue Links:
Related
is related to SERVER-23925 Potential use-after-free when WSM is ... Closed
Participants:

 Description   

Was suggested by acm in SERVER-23925. I attempted to run a builder with the following definition, but ran into linker errors.

diff --git a/etc/evergreen.yml b/etc/evergreen.yml
index 45ff93f..bdc76d8 100644
--- a/etc/evergreen.yml
+++ b/etc/evergreen.yml
@@ -224,7 +224,7 @@ functions:
         rm -rf rocksdb
         git clone https://github.com/facebook/rocksdb.git
         cd rocksdb
-        make CC=/opt/mongodbtoolchain/v2/bin/gcc CXX=/opt/mongodbtoolchain/v2/bin/g++ static_lib
+        make ${compile_flags_rocksdb} static_lib
         fi
 
   "build new tools" :
@@ -5832,7 +5832,8 @@ buildvariants:
   - name: tool
   - name: unittests
 
-- name: ubuntu1404-rocksdb
+- &rocksdb_variant
+  name: ubuntu1404-rocksdb
   display_name: Ubuntu 14.04 (RocksDB)
   modules:
   - rocksdb
@@ -5841,6 +5842,7 @@ buildvariants:
   batchtime: 1440 # 1 day
   expansions:
     build_rocksdb: true
+    compile_flags_rocksdb: CC=/opt/mongodbtoolchain/v2/bin/gcc CXX=/opt/mongodbtoolchain/v2/bin/g++
     compile_flags: -j$(grep -c ^processor /proc/cpuinfo) --dbg=off --opt=on CPPPATH=$(readlink -f ../rocksdb/include/) LIBPATH=$(readlink -f ../rocksdb/) LIBS=rocksdb CC=/opt/mongodbtoolchain/v2/bin/gcc CXX=/opt/mongodbtoolchain/v2/bin/g++ VARIANT_DIR=release
     num_jobs_available: $(grep -c ^processor /proc/cpuinfo)
     test_flags: --storageEngine=rocksdb
@@ -6021,6 +6023,20 @@ buildvariants:
   - name: tool_WT
   - name: unittests
 
+- <<: *rocksdb_variant
+  name: ubuntu1410-rocksdb-asan
+  display_name: ~ ASAN Ubuntu 14.10 (RocksDB)
+  run_on:
+  - ubuntu1410-build
+  expansions:
+    build_rocksdb: true
+    compile_flags_rocksdb: COMPILE_WITH_ASAN=1 CC=/usr/bin/clang CXX=/usr/bin/clang++
+    san_symbolizer: ASAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-3.5
+    san_options: LSAN_OPTIONS="suppressions=etc/lsan.suppressions" ASAN_OPTIONS=detect_leaks=1
+    compile_flags: CPPPATH=$(readlink -f ../rocksdb/include/) LIBPATH=$(readlink -f ../rocksdb/) LIBS=rocksdb CC=/usr/bin/clang CXX=/usr/bin/clang++ --opt=on --allocator=system --sanitize=address --ssl --libc++ -j$(grep -c ^processor /proc/cpuinfo) --nostrip VARIANT_DIR=build
+    num_jobs_available: $(grep -c ^processor /proc/cpuinfo)
+    test_flags: --storageEngine=rocksdb -j1 # Avoid starting too many mongod's under ASAN build.
+
 - name: ubuntu1410-asan
   display_name: ~ ASAN SSL Ubuntu 14.10
   run_on:

Patch to apply to mongodb-partners/mongo-rocks

The ASSERT*() macros require that the arguments are ostream-able. It isn't clear why this only fails when compiling with clang and not gcc.

diff --git a/src/rocks_record_store_test.cpp b/src/rocks_record_store_test.cpp
index a90ada2..ac19191 100644
--- a/src/rocks_record_store_test.cpp
+++ b/src/rocks_record_store_test.cpp
@@ -29,6 +29,7 @@
 #include "mongo/platform/basic.h"
 
 #include <boost/filesystem/operations.hpp>
+#include <boost/optional/optional_io.hpp>
 #include <memory>
 #include <vector>
 



 Comments   
Comment by Max Hirschhorn [ 26/Sep/18 ]

We've removed the RocksDB build variant in SERVER-35859 that adding another variation of it running with ASan wouldn't make sense.

Comment by Max Hirschhorn [ 02/Dec/16 ]

There is no goal of adding a regular ASAN RocksDB builder since ASAN capable machine resources are very limited.

My impression is that there'd still be value in having a RocksDB+ASan build variant running in Evergreen given that it tends to be more aggressive about freeing memory returned from the storage engine after subsequent calls to the cursor. This leads to issues such as SERVER-23925 being more readily tracked down when using RocksDB+ASan.

Any returned unowned BSON is only valid until the next call to any method on this interface.

If we aren't actually interested in periodically running this build variant in Evergreen, then I think we should strongly consider adding a wrapper for SortedDataInterface::Cursor and RecordCursor that conditionally frees the memory previously returned on subsequent function calls, as was effectively done to the WiredTiger integration layer via the patch from this comment. We could then enable that mode when running our concurrency tests (and possibly others). This would help to ensure that the query subsystem is using the storage engine in a way that doesn't violate the API contract.

Comment by Alexander Gorrod [ 29/Nov/16 ]

We have been able to create ASAN patch builds with the RocksDB storage engine in Evergreen. There is no goal of adding a regular ASAN RocksDB builder since ASAN capable machine resources are very limited. The diff file used to generate an ASAN build is attached to this ticket.

Comment by Igor Canadi [ 02/May/16 ]

Hi Max, thanks for trying this out.

Looks like the cause of compile errors could be the difference in rocks and mongo's libc++.
When you link with RocksDB it's trying to get symbols from its version of libc++, but it's different from the one you're linking with.

Can you please try two things:
1. Remove --libc++ from the compile_flags. I think that's causing mongo to compile with different version of libc++ thus causing link failures.
2. Try compiling mongod with g++ (i.e. don't change 'compile_flags_rocksdb'). g++ since version 4.8 also supports ASAN sanitizer (see https://en.wikipedia.org/wiki/AddressSanitizer)

Also, can you please add V=1 to the 'compile_flags_rocksdb' (v as in 'verbose')? That will cause our compile process to output exact compiler invocation command.

Generated at Thu Feb 08 04:05:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.