-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Unknown
-
None
-
Affects Version/s: 2.5.4
-
Not Needed
-
None
-
C Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Problem
_mongoc_topology_scanner_append_metadata() (reached from mongoc_client_append_metadata() and mongoc_client_pool_append_metadata()) copies the global handshake struct by value:
// src/libmongoc/src/mongoc/mongoc-topology-scanner.c:749 (2.5.4; unchanged on master)
mongoc_handshake_t md = *_mongoc_handshake_get();
mongoc_handshake_t contains int8_t frozen, an atomic flag that _mongoc_handshake_freeze() (mongoc-handshake.c:829) writes with mcommon_atomic_int8_exchange. That freeze runs from _initialize_handshake_cmd() (mongoc-topology-scanner.c:380) every time a client builds its handshake command for the first time, not once per process. The plain memcpy read of frozen in one thread and the atomic write in another are a data race: benign in practice (the copied value is never used), but undefined behaviour under the C11 memory model, and it aborts ThreadSanitizer builds that run with halt_on_error.
How it was found
ClickHouse embeds libmongoc via mongocxx and started calling mongocxx::client::append_metadata() for every client (ClickHouse/ClickHouse#122205). Its TSAN CI aborts when several MongoDB-backed dictionaries are reloaded in parallel (trimmed):
WARNING: ThreadSanitizer: data race
Atomic write of size 1 at 0x563c56ab1080 by thread T199:
#0 mcommon_atomic_int8_exchange common-atomic-private.h:331
#1 _mongoc_handshake_freeze mongoc-handshake.c:829
#2 _initialize_handshake_cmd mongoc-topology-scanner.c:380
#3 _mongoc_topology_scanner_append_metadata mongoc-topology-scanner.c:651
#4 mongoc_client_append_metadata mongoc-client.c:2844
Previous read of size 8 at 0x563c56ab1080 by thread T189 (mutexes: write M0):
#0 __tsan_memcpy
#1 _mongoc_topology_scanner_append_metadata mongoc-topology-scanner.c:749
#2 mongoc_client_append_metadata mongoc-client.c:2844
Location is global 'gMongocHandshake' of size 112 at 0x563c56ab1018
M0 is the per-client ts->handshake_cmd_mtx, so it does not order the two threads.
Reproducer (no server needed)
mongoc_client_append_metadata() never connects. Build libmongoc and this program with -fsanitize=thread; it reports the race above on every run (macOS 15, Apple clang 21, libmongoc 2.5.4). On macOS run it with TSAN_OPTIONS=ignore_interceptors_accesses=0, because Darwin TSAN ignores accesses made inside interceptors such as memcpy by default and hides the race.
#include <mongoc/mongoc.h> #include <pthread.h> #include <stdio.h> #define NTHREADS 16 #define ROUNDS 50 /* Minimal reusable barrier (macOS has no pthread_barrier_t). */ static pthread_mutex_t barrier_mtx = PTHREAD_MUTEX_INITIALIZER; static pthread_cond_t barrier_cv = PTHREAD_COND_INITIALIZER; static int barrier_arrived = 0; static int barrier_generation = 0; static void barrier_wait (void) { pthread_mutex_lock (&barrier_mtx); int gen = barrier_generation; if (++barrier_arrived == NTHREADS) { barrier_arrived = 0; barrier_generation++; pthread_cond_broadcast (&barrier_cv); } else { while (gen == barrier_generation) { pthread_cond_wait (&barrier_cv, &barrier_mtx); } } pthread_mutex_unlock (&barrier_mtx); } static void * thread_fn (void *arg) { const int id = (int) (intptr_t) arg; for (int round = 0; round < ROUNDS; round++) { mongoc_client_t *client = mongoc_client_new ("mongodb://localhost:27017"); barrier_wait (); /* Stagger the threads (busy-wait, so no sleep-based ordering is involved): * thread 0 reaches the struct copy while later threads are still about * to run _mongoc_handshake_freeze(). */ for (volatile long spin = 0; spin < (long) id * 20000; spin++) { } if (!mongoc_client_append_metadata (client, "repro", "1.0", NULL)) { fprintf (stderr, "mongoc_client_append_metadata failed\n"); } mongoc_client_destroy (client); } return NULL; } int main (void) { pthread_t threads[NTHREADS]; mongoc_init (); for (int i = 0; i < NTHREADS; i++) { pthread_create (&threads[i], NULL, thread_fn, (void *) (intptr_t) i); } for (int i = 0; i < NTHREADS; i++) { pthread_join (threads[i], NULL); } mongoc_cleanup (); return 0; }
cmake -S mongo-c-driver -B build -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_FLAGS="-fsanitize=thread -g" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread" -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=thread" \
-DENABLE_TESTS=OFF -DENABLE_EXAMPLES=OFF -DENABLE_SSL=OFF -DENABLE_SASL=OFF -DENABLE_ZLIB=OFF -DENABLE_SNAPPY=OFF -DENABLE_ZSTD=OFF -DENABLE_SRV=OFF \
-DCMAKE_INSTALL_PREFIX=$PWD/prefix
cmake --build build -j && cmake --install build
clang -fsanitize=thread -g append_metadata_race.c -I prefix/include/mongoc-2.5.4 -I prefix/include/bson-2.5.4 \
-L prefix/lib -lmongoc2 -lbson2 -Wl,-rpath,$PWD/prefix/lib -o append_metadata_race
TSAN_OPTIONS=ignore_interceptors_accesses=0 ./append_metadata_race
Note: a naive tight loop of create/append/destroy does not reproduce it. The atomic OID counter in mongoc_client_new() and the acquire load in _mongoc_handshake_get() create happens-before edges that mask the race, hence the barrier and stagger above.
- found in
-
ClickHouse/ClickHouse#122205: Identify ClickHouse in the MongoDB driver handshake
ClickHouse PR whose TSAN CI hits this race