Priority: Major - P3
Affects Version/s: None
Fix Version/s: None
We currently implement wrappers around malloc and realloc (called mongoMalloc and mongoRealloc, respectively), which terminate the process on allocation failure. Similarly, we set a new_handler to accomplish the same tasks for the C++ memory interface.
Unfortunately, we don't currently have something in place to make allocation failure in generic third_party code hard fail on allocation failure. If, say, s2 made a call to malloc and failed to properly handle the NULL return case, we are likely to see a crash at best or tricky memory corruption at worst.
Our vendored tcmalloc offers a facility called tc_new_mode, which changes the semantics of the C allocation routines so that they call new_handler instead of returning NULL. For a case like ours where new_handler is known to terminate, this would work well.
But this isn't a perfect solution: on some platforms we build with --allocator=system. For those platforms, we can't rely on tc_new_mode. For windows, we could investigate using the windows analogue of tc_new_mode, which is called _set_new_mode. For the cases where Windows directly calls into tcmalloc, we could continue to use tc_set_new_mode.
That still leaves macOS (where we build with --allocator=system, and the issue still exists for anyone doing community builds with --allocator=system.
The best choice is probably for us to stop relying on tcmalloc's C and C++ allocation interface interceptors, and to write our own. We would inject them using the same techniques that tcmalloc itself uses (linker magic, etc.). Then we would backstop them with either plan calls to the basic C library routines, or we would call into the appropriate tc_ routines if we were using tcmalloc.
We could then remove the definitions of mongoMalloc and mongoRealloc, since we would obtain equivalent behavior via our injected implementations.