[SERVER-50125] UBSAN with ccache doesn't always respect -fsanitize-blacklist options Created: 05/Aug/20  Updated: 29/Oct/23  Resolved: 03/Sep/20

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 4.7.0, 4.4.2

Type: Bug Priority: Major - P3
Reporter: Spencer Jackson Assignee: Ryan Egesdahl (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Dev Platform 2020-08-24, Dev Platform 2020-09-07
Participants:

 Description   

When I build the server on my evergreen workstation with AUBSAN, it crashes with messages like the following:

{"t":{"$date":"2020-08-05T14:10:35.751+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
src/third_party/murmurhash3/MurmurHash3.cpp:92:49: runtime error: addition of unsigned offset to 0x7ffe6ba480d0 overflowed to 0x7ffe6ba480c0
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/third_party/murmurhash3/MurmurHash3.cpp:92:49 in

I've also seen messages from Snappy. These errors are surprising, as they're in files covered by etc/ubsan.blacklist which I would expect to suppress them. On a hunch, I tried disabling ccache, and the resulting mongod binaries appear to work. I found this Github issue which appears to describe the problem: https://github.com/ccache/ccache/issues/174 It was tagged as being a part of their 3.5 epic. The version of ccache on the evergreen workstations appears to be version 3.4.1.

The SCons invokation which produced the failing binaries:

python ./buildscripts/scons.py --dbg=on --opt=on --variables-files=etc/scons/mongodbtoolchain_v3_clang.vars --allocator=system --sanitize=address,undefined --ninja=next generate-ninja MONGO_VERSION="0.0.0" MONGO_GIT_HASH="unknown" CCACHE=ccache

The corrected invokation:

python ./buildscripts/scons.py --dbg=on --opt=on --variables-files=etc/scons/mongodbtoolchain_v3_clang.vars --allocator=system --sanitize=address,undefined --ninja=next generate-ninja MONGO_VERSION="0.0.0" MONGO_GIT_HASH="unknown"



 Comments   
Comment by Githook User [ 13/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e7f6696a96db5438a269c5e67a06ff9e05cb897a

Comment by Githook User [ 07/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

(cherry picked from commit 78bb3f3c8a658a5a9fec8d55864e426382f68bd0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/7b93770e0835087c023ae3e3f36eeb46d204767f

Comment by Githook User [ 07/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50125 Add sanitizer blacklist files to CCACHE_EXTRAFILES

Some versions of ccache do not know how to handle clang's
-fsanitizer-blacklist flags. Some versions don't handle it at all, while
others only handle one instance, even though it can appear multiple
times on the command line. Because the argument can change the resulting
compiled object, not taking the flags into account properly can cause
ccache to pull an incorrect object file from its cache. The exact
behavior depends on the ccache version and how the arguments are changed
on the command line. We implement a workaround suggested by the ccache
developers until a newer version of ccache with all the required fixes
is in common use.

(cherry picked from commit 37276b21f4dbd66f913e8d49577fd4b1c4eafbf9)
Branch: v4.4
https://github.com/mongodb/mongo/commit/3a3cb2c81939d0acba655890c586351d122673de

Comment by Githook User [ 06/Oct/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50363 Merge --build-tools=next into stable

Merging the following fixes into the stable version of the build tools
and migrate fully from --ninja=next to the --build-tools=next:

Comment by Githook User [ 03/Sep/20 ]

Author:

{'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}

Message: SERVER-50125 Add sanitizer blacklist files to CCACHE_EXTRAFILES

Some versions of ccache do not know how to handle clang's
-fsanitizer-blacklist flags. Some versions don't handle it at all, while
others only handle one instance, even though it can appear multiple
times on the command line. Because the argument can change the resulting
compiled object, not taking the flags into account properly can cause
ccache to pull an incorrect object file from its cache. The exact
behavior depends on the ccache version and how the arguments are changed
on the command line. We implement a workaround suggested by the ccache
developers until a newer version of ccache with all the required fixes
is in common use.

Comment by Ryan Egesdahl (Inactive) [ 05/Aug/20 ]

ccache added support for hashing -fsanitize-blacklist in version 3.5 with this PR:
https://github.com/ccache/ccache/pull/258

There is still a pending issue open to handle multiple -fsanitize-blacklist options as well:
https://github.com/ccache/ccache/issues/318

The -fsanitize-blackist option changes the resulting binary object output by the compiler. These missing options mean that someone using ccache who has compiled without -fsanitize-blacklist and then with it (or vice/versa) may see the wrong object being pulled from the cache. For the record, Ubuntu 18.04 currently still uses ccache version 3.4.1, so it doesn't support hashing -fsanitize-blacklist at all. Ubuntu 20.04 has version 3.7.7, so it supports only one instance of -fsanitize-blacklist. There is a workaround documented in the issue associated with PR #258 that we might be able to use here:
https://github.com/ccache/ccache/issues/174

Basically, we just add the paths to the blacklist files to the CCACHE_EXTRAFILES environment variable so ccache alters the hash according to whether they are in use. This should work regardless of the ccache version.

Generated at Thu Feb 08 05:21:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.