ExportXMLWordPrintableJSON

    • Not Needed
    • None
    • C Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      _mongoc_topology_scanner_append_metadata() (reached from mongoc_client_append_metadata() and mongoc_client_pool_append_metadata()) copies the global handshake struct by value:

      // src/libmongoc/src/mongoc/mongoc-topology-scanner.c:749 (2.5.4; unchanged on master)
      mongoc_handshake_t md = *_mongoc_handshake_get();
      

      mongoc_handshake_t contains int8_t frozen, an atomic flag that _mongoc_handshake_freeze() (mongoc-handshake.c:829) writes with mcommon_atomic_int8_exchange. That freeze runs from _initialize_handshake_cmd() (mongoc-topology-scanner.c:380) every time a client builds its handshake command for the first time, not once per process. The plain memcpy read of frozen in one thread and the atomic write in another are a data race: benign in practice (the copied value is never used), but undefined behaviour under the C11 memory model, and it aborts ThreadSanitizer builds that run with halt_on_error.

      How it was found

      ClickHouse embeds libmongoc via mongocxx and started calling mongocxx::client::append_metadata() for every client (ClickHouse/ClickHouse#122205). Its TSAN CI aborts when several MongoDB-backed dictionaries are reloaded in parallel (trimmed):

      WARNING: ThreadSanitizer: data race
        Atomic write of size 1 at 0x563c56ab1080 by thread T199:
          #0 mcommon_atomic_int8_exchange common-atomic-private.h:331
          #1 _mongoc_handshake_freeze mongoc-handshake.c:829
          #2 _initialize_handshake_cmd mongoc-topology-scanner.c:380
          #3 _mongoc_topology_scanner_append_metadata mongoc-topology-scanner.c:651
          #4 mongoc_client_append_metadata mongoc-client.c:2844
        Previous read of size 8 at 0x563c56ab1080 by thread T189 (mutexes: write M0):
          #0 __tsan_memcpy
          #1 _mongoc_topology_scanner_append_metadata mongoc-topology-scanner.c:749
          #2 mongoc_client_append_metadata mongoc-client.c:2844
        Location is global 'gMongocHandshake' of size 112 at 0x563c56ab1018
      

      M0 is the per-client ts->handshake_cmd_mtx, so it does not order the two threads.

      Reproducer (no server needed)

      mongoc_client_append_metadata() never connects. Build libmongoc and this program with -fsanitize=thread; it reports the race above on every run (macOS 15, Apple clang 21, libmongoc 2.5.4). On macOS run it with TSAN_OPTIONS=ignore_interceptors_accesses=0, because Darwin TSAN ignores accesses made inside interceptors such as memcpy by default and hides the race.

      #include <mongoc/mongoc.h>
      #include <pthread.h>
      #include <stdio.h>
      
      #define NTHREADS 16
      #define ROUNDS 50
      
      /* Minimal reusable barrier (macOS has no pthread_barrier_t). */
      static pthread_mutex_t barrier_mtx = PTHREAD_MUTEX_INITIALIZER;
      static pthread_cond_t barrier_cv = PTHREAD_COND_INITIALIZER;
      static int barrier_arrived = 0;
      static int barrier_generation = 0;
      
      static void
      barrier_wait (void)
      {
         pthread_mutex_lock (&barrier_mtx);
         int gen = barrier_generation;
         if (++barrier_arrived == NTHREADS) {
            barrier_arrived = 0;
            barrier_generation++;
            pthread_cond_broadcast (&barrier_cv);
         } else {
            while (gen == barrier_generation) {
               pthread_cond_wait (&barrier_cv, &barrier_mtx);
            }
         }
         pthread_mutex_unlock (&barrier_mtx);
      }
      
      static void *
      thread_fn (void *arg)
      {
         const int id = (int) (intptr_t) arg;
      
         for (int round = 0; round < ROUNDS; round++) {
            mongoc_client_t *client = mongoc_client_new ("mongodb://localhost:27017");
      
            barrier_wait ();
            /* Stagger the threads (busy-wait, so no sleep-based ordering is involved):
             * thread 0 reaches the struct copy while later threads are still about
             * to run _mongoc_handshake_freeze(). */
            for (volatile long spin = 0; spin < (long) id * 20000; spin++) {
            }
      
            if (!mongoc_client_append_metadata (client, "repro", "1.0", NULL)) {
               fprintf (stderr, "mongoc_client_append_metadata failed\n");
            }
      
            mongoc_client_destroy (client);
         }
         return NULL;
      }
      
      int
      main (void)
      {
         pthread_t threads[NTHREADS];
      
         mongoc_init ();
         for (int i = 0; i < NTHREADS; i++) {
            pthread_create (&threads[i], NULL, thread_fn, (void *) (intptr_t) i);
         }
         for (int i = 0; i < NTHREADS; i++) {
            pthread_join (threads[i], NULL);
         }
         mongoc_cleanup ();
         return 0;
      }
      
      cmake -S mongo-c-driver -B build -DCMAKE_BUILD_TYPE=Debug \
            -DCMAKE_C_FLAGS="-fsanitize=thread -g" -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread" -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=thread" \
            -DENABLE_TESTS=OFF -DENABLE_EXAMPLES=OFF -DENABLE_SSL=OFF -DENABLE_SASL=OFF -DENABLE_ZLIB=OFF -DENABLE_SNAPPY=OFF -DENABLE_ZSTD=OFF -DENABLE_SRV=OFF \
            -DCMAKE_INSTALL_PREFIX=$PWD/prefix
      cmake --build build -j && cmake --install build
      clang -fsanitize=thread -g append_metadata_race.c -I prefix/include/mongoc-2.5.4 -I prefix/include/bson-2.5.4 \
            -L prefix/lib -lmongoc2 -lbson2 -Wl,-rpath,$PWD/prefix/lib -o append_metadata_race
      TSAN_OPTIONS=ignore_interceptors_accesses=0 ./append_metadata_race
      

      Note: a naive tight loop of create/append/destroy does not reproduce it. The atomic OID counter in mongoc_client_new() and the acquire load in _mongoc_handshake_get() create happens-before edges that mask the race, hence the barrier and stagger above.

       

            Assignee:
            Unassigned
            Reporter:
            Rishabh Bisht
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: