The current API for loading extensions relies on a single function (get_mongodb_extension), which is guaranteed to be stable starting from mongodb version 9.0. get_mongodb_extension is the single point of handshake between the host and the extension at load time. The host provides the extension with the major versions the host supports, and it is up to the extension to return a compatible extension version back to the host.

Unfortunately, this API has some shortcomings that we did not account for when we initially decided on it. At the time, we had not yet identified the requirements for the extension backwards compatibility story. 

The first point, is that get_mongodb_extension accepts a HostServicesAPI, through which logging and error reporting can be accessed by the extension. Part of the API's contract is that all errors and logging must be generated using the HostServicesAPI. The problem is that get_mongodb_extension is responsible for version negotiation, meaning that until a compatible version is agreed upon between the host and the extension, it is not safe or valid for the extension to use any APIs provided by the Host. For example, if the Host is on API major version 3, and it tries to load an extension from a shared object which only supports major versions 1 and 2, the Host would have provided the extension with a HostServicesAPI which the extension may not understand at all, as the API may have changed across the major versions.

Recently, we encountered issues during 8.3 roll-out of disagg, where the $rerank extension they had been testing with was built a version that did not match the server's. The logs available were not particularly helpful, since the logs simply stated that the extension failed to return a valid pointer back to the host. Some considerable investigation effort was required to figure out what had caused the version mismatch in the extension binaries. This highlights the importance of having more descriptive logging during the extension loading to easily identify the root cause of extension loading issues.

To this end, we propose changing the extension loading procedure into a two step process. We will introduce one more symbol to the api header, which must be implemented by every extension:
get_mongodb_extension_versions. 

Extension loading will be a two-step process driven by the host across two symbols that the extension shared library must export:
1. get_mongodb_extension_versions: the host asks the extension to specify all the versions of the Extensions API the extension implements.
2. get_mongodb_extension: the host deliberates on whether any of those published versions are compatible with its own supported set. If a compatible version is found, the host explicitly requests that version by invoking this function with the chosen version. During this step, the host provides the HostServicesAPI corresponding to the requested API version.

This change in procedure allows us to move the version negotiation entirely to the Host. This allows us to have more verbose & descriptive logging when an extension can't be loaded by making the extension's versions available to the host directly. Furthermore, this removes the complexity around providing a compatible HostServicesAPI to the extension during the loading process. Once the host finds a compatible version, it provides the correctly versioned HostServicesAPI to the extension when requesting the compatible version from the extension. 

We think the added complexity of the additional step during extension loading is justified by the improvements to the APIs long term stability and improved error reporting.