[CSHARP-4889] Support Linux distros using libdl.so.2 Created: 07/Feb/23  Updated: 26/Jan/24

Status: Backlog
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Eric Rosenquist Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: rp-track
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-49140 [FLE] Add mongocryptd for Linux Alpine Closed
is related to CSHARP-4363 Investigate csfle assembly loading fa... Blocked
Quarter: FY25Q1
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

CSFLE with the .NET driver doesn't work on Linux distros such as Debian 11 due to a dependency on libdl.so for loading the libmongocrypt.so library. On Debian 11 (and several other distros), the dlopen and dlsym functions reside in libdl.so.2. The dependency is in the LinuxLibrary class in bindings/cs/MongoDB.Libmongocrypt/LibraryLoader.cs

At runtime, the library throws an exception such as:

System.DllNotFoundException: Unable to load shared library 'libdl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: liblibdl: cannot open shared object file: No such file or directory
   at MongoDB.Libmongocrypt.LibraryLoader.LinuxLibrary.dlopen(String filename, Int32 flags)
   at MongoDB.Libmongocrypt.LibraryLoader.LinuxLibrary..ctor(String path)
   at MongoDB.Libmongocrypt.LibraryLoader..ctor()
   at MongoDB.Libmongocrypt.Library.<>c.<.cctor>b__0_65()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at MongoDB.Libmongocrypt.Library.<>c.<.cctor>b__0_1()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at MongoDB.Libmongocrypt.CryptClientFactory.Create(CryptOptions options)
   at MongoDB.Driver.Core.Clusters.CryptClientCreator.CreateCryptClient(CryptOptions options)
   at MongoDB.Driver.Core.Clusters.CryptClientCreator.CreateCryptClient(CryptClientSettings cryptClientSettings)
   at MongoDB.Driver.Encryption.ClientEncryption..ctor(ClientEncryptionOptions clientEncryptionOptions)

The NativeLibrary class might help, but can't be used since libmongocrypt targets .NET Standard and that library is not available on .NET Standard.

The https://github.com/mellinoe/nativelibraryloader package might be a viable alternative since it supports .NET Standard 2.0 and supports systems that use libdl.so as well as ones that use libdl.so.2



 Comments   
Comment by Vishal Rastogi [ 18/Dec/23 ]

We've been using Atlas + CSFLE for the last 2 years and have constantly been struggling with this issue. Now with Net8, even though libdl.so.2 is available in the containers out of the box we're having to do various hacks to get this working. Given FLE is an enterprise feature, this should just work with the official dotnet docker images. There is absolutely no documentation around this and all documentation, if any, is dated and buried deep inside.

Comment by James Kovacs [ 22/Nov/23 ]

Given that our current dlopen dynamic loading technique fails on certain Linux distros such as Debian 11 and Alpine, we are going to investigate options for dynamic loading that are more cross-Linux compatible. Please follow this ticket for updates.

Comment by Vasistan Shakkaravarthi [ 15/Nov/23 ]

Hi Team, Do we have any updates on this? It looks like this feature is still in an unstable state. I recently got the chance to adopt this feature with one of our works but ended up with a dead-end sort of status. Because this is not stable enough in .NET 6 container images. We have created a support ticket related to this https://support.mongodb.com/case/01221814

Comment by Dmitry Lukyanov (Inactive) [ 28/Apr/23 ]

james.kovacs@mongodb.com I've reassigned this on you to not lose this ticket

Comment by Dmitry Lukyanov (Inactive) [ 28/Apr/23 ]

The easier solution I see is to create different "ISharedLibraryLoader"s impls on OSs where methods "dlopen", "dlsym" can be placed into different libraries. So, for linux we will need at least 2 with "libdl" impl and "libdl.so.2". And then here we will need to validate whether underlying library loader call succeeded or no. If no and if we have a different loader, try creating it.

The problem here is configuring EG env. Given that Debian11 doesn't have .net installed, we need to prepare it beforehand. The easiest way is to add cake scripts from the driver, but it didn't work well for me (see my branch).

Comment by Eric Rosenquist [ 15/Feb/23 ]

That would be a reference to libdl.so.2 that was automatically added during linking. If you do nm -g libmongocrypt.so | grep dl you can see that symbols like dlopen actually come from GLIBC. That's expected since libdl.so* is now basically an empty stub library and the routines are in libc.

Regardless of that, this bug is because the C# code uses DllImport in bindings/cs/MongoDB.Libmongocrypt/LibraryLoader.cs to find the dlopen and dlsym functions around line 206. The .NET DllImport implementation will add lib as a prefix and .so as a suffix if necessary, but it doesn't know that some Linux distros use libdl.so and others use libdl.so.2 - so in this case it just looks for libdl.so.

I'm running code in a Debian 11.6 container based on mcr.microsoft.com/dotnet/aspnet:6.0, which is based on amd64/debian:bullseye-slim

For what it's worth, using DllImport to find dlopen and dlsym to bind functions is normally not needed in .NET, which is why other libraries don't run into similar issues. If you have a native library and you want to use it in .NET, you typically just declare a .NET function for each entry point using DllImport and you specify the library name to DllImport. An entry point like mongocrypt_status_new would have a C# binding that looks something like:

    [DllImport("mongocrypt", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
    internal static extern StatusSafeHandle mongocrypt_status_new();

The paths that DllImport searches can be controlled with the DefaultDllImportSearchPaths attribute, but I doubt you even need that.

Comment by Roberto Sanchez [ 15/Feb/23 ]

I have looked at the libmongocrypt.so runtime library at the link you provided. According to readelf -a ./libmongocrypt.so it wants libdl.so.2:

0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]

When I run ldd on the shared library it always finds libdl.so.2. I verified this in Debian environments from Debian 7 all the way to a current Debian unstable.

Can you provide a specific case, including OS version, and steps to reproduce the fault you are experiencing?

Comment by Eric Rosenquist [ 14/Feb/23 ]

It's bundled with the .NET driver. The main .NET driver is MongoDB.Driver which depends on MongoDB.Libmongocrypt.. The latter bundles Windows, Linux, and macOS copies of libmongocrypt which you can see under the runtimes folder at https://nuget.info/packages/MongoDB.Libmongocrypt/1.7.0

Comment by Roberto Sanchez [ 13/Feb/23 ]

eric_rosenquist@cysiv.com, can you provide some information about how you are building libmongocrypt, or where you are obtaining packages (in the case that you are using binary packages)?

Generated at Wed Feb 07 21:49:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.