Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63710

CSFLE: LeakSanitizer warnings after unloading the dynamic library

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Field Level Encryption
    • Labels:
    • Server Security
    • ALL

      On Linux, if an application dlopens the csfle dynamic library then dlcloses the library, LeakSanitizer later warns about memory leaks during program exit. Omitting the call to dlclose (or opening with RTLD_NODELETE) causes the leaks to vanish. No library APIs need be invoked for the behavior to show.

      LeakSanitizer is unable to render the symbols, unfortunately, as the dynamic library has been unloaded from the memory mappings before LeakSanitizer catches the issue at program shutdown. Our best guess is that this is most likely leaking in statically constructed objects that are built up during dlopen as part of library initialization, but those same objects are never destroyed as part of dlclose. This has a visible effect in LeakSanitizer, but means that other possible important side-effects of static destructors will also never occur.

      Possible hint: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71971 which relates to a quirk of ELF and ld-linux when static symbols are annotated with UNIQUE, however no symbols in the library appear to have that annotation. I can verify that the call of dlopen transitively invokes __cxa_atexit to register static destructors, but I do not know why they do not run on dlclose.

      Because dlopen is smart enough not to allocate duplicate mappings in case of repeated dlopen of the same library, it is safe to never close the library and let the static destructors run at program shutdown. This is a viable workaround unless we need to rely on static destructors running immediately on dlclose.

      The following code is enough to reproduce the issue:

      #include <dlfcn.h>
      
      #include <iostream>
      
      int
      main ()
      {
         void *lib = ::dlopen ("/path/to/mongo_csfle_v1.so", RTLD_NOW | RTLD_GLOBAL);
         if (lib == nullptr) {
            std::cerr << "dlopen() failed: " << ::dlerror () << '\n';
            return 2;
         }
         ::dlclose (lib);
         return 0;
      }
      

            Assignee:
            backlog-server-security [DO NOT USE] Backlog - Security Team
            Reporter:
            colby.pike@mongodb.com Colby Pike
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: