[SERVER-1266] libmongoclient should not link with xulrunner's libmozjs, causes subtle memory errors Created: 21/Jun/10  Updated: 12/Jul/16  Resolved: 26/Jul/10

Status: Closed
Project: Core Server
Component/s: Internal Client
Affects Version/s: 1.5.3
Fix Version/s: 1.5.7

Type: Improvement Priority: Major - P3
Reporter: Andrew Morrow (Inactive) Assignee: Michael Dirolf
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 10.04 x86_64


Issue Links:
Depends
depends on SERVER-1355 separate c++ client tarball Closed
Participants:

 Description   

An empty mainline, when linked with libmongoclient, then run under valgrind, reports a bad free at shutdown in __libc_freeres processing:

mongo_libc_freeres$ cat ./main.cc
#include <cstdio>

int main(int argc, char* argv[])
{
std::printf("Hello, World\n");
return 0;
}

mongo_libc_freeres$ g++ -ggdb ./main.cc -L /mongo.temp.install/lib64 -Wl,-rpath,/mongo.temp.install/lib64 -lmongoclient

mongo_libc_freeres$ ls -la ./a.out
-rwxrwxr-x 1 xxx xxx 12666 2010-06-21 12:08 ./a.out

mongo_libc_freeres$ ldd ./a.out
linux-vdso.so.1 => (0x00007fffc50ce000)
libmongoclient.so.9.9.9 => ~/mongo.temp.install/lib64/libmongoclient.so.9.9.9 (0x00007f20cb1cb000)
libstdc+.so.6 => /usr/lib/libstdc+.so.6 (0x00007f20cae97000)
libm.so.6 => /lib/libm.so.6 (0x00007f20cac13000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f20ca9fc000)
libc.so.6 => /lib/libc.so.6 (0x00007f20ca67a000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f20ca45c000)
libboost_system.so.1.40.0 => /usr/lib/libboost_system.so.1.40.0 (0x00007f20ca258000)
libboost_thread.so.1.40.0 => /usr/lib/libboost_thread.so.1.40.0 (0x00007f20ca042000)
libboost_filesystem.so.1.40.0 => /usr/lib/libboost_filesystem.so.1.40.0 (0x00007f20c9e2c000)
libboost_program_options.so.1.40.0 => /usr/lib/libboost_program_options.so.1.40.0 (0x00007f20c9bde000)
libpcrecpp.so.0 => /usr/lib/libpcrecpp.so.0 (0x00007f20c99d5000)
libpcre.so.3 => /lib/libpcre.so.3 (0x00007f20c97a6000)
libmozjs.so => /usr/lib64/xulrunner-1.9.2.3/libmozjs.so (0x00007f20c948d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f20cb6a3000)
librt.so.1 => /lib/librt.so.1 (0x00007f20c9284000)
libplds4.so => /usr/lib/libplds4.so (0x00007f20c9080000)
libplc4.so => /usr/lib/libplc4.so (0x00007f20c8e7a000)
libnspr4.so => /usr/lib/libnspr4.so (0x00007f20c8c3e000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f20c8a3a000)

mongo_libc_freeres$ valgrind --track-origins=yes --read-var-info=yes --leak-check=full ./a.out
==29614== Memcheck, a memory error detector
==29614== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==29614== Using Valgrind-3.6.0.SVN and LibVEX; rerun with -h for copyright info
==29614== Command: ./a.out
==29614==
Hello, World
==29614== Invalid free() / delete / delete[]
==29614== at 0x4C28D5E: free (vg_replace_malloc.c:381)
==29614== by 0x5BE404A: free_mem (in /lib/libc-2.11.1.so)
==29614== by 0x5BE3BE1: __libc_freeres (in /lib/libc-2.11.1.so)
==29614== by 0x4A236AB: _vgnU_freeres (vg_preloaded.c:62)
==29614== by 0x5AEC214: exit (exit.c:93)
==29614== by 0x5AD1C53: (below main) (libc-start.c:258)
==29614== Address 0x4043e88 is not stack'd, malloc'd or (recently) free'd
==29614==
==29614==
==29614== HEAP SUMMARY:
==29614== in use at exit: 10,512 bytes in 25 blocks
==29614== total heap usage: 421 allocs, 397 frees, 40,988 bytes allocated
==29614==
==29614== LEAK SUMMARY:
==29614== definitely lost: 0 bytes in 0 blocks
==29614== indirectly lost: 0 bytes in 0 blocks
==29614== possibly lost: 0 bytes in 0 blocks
==29614== still reachable: 10,512 bytes in 25 blocks
==29614== suppressed: 0 bytes in 0 blocks
==29614== Reachable blocks (those to which a pointer was found) are not shown.
==29614== To see them, rerun with: --leak-check=full --show-reachable=yes
==29614==
==29614== For counts of detected and suppressed errors, rerun with: -v
==29614== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 6 from 6)

If we manually re-link libmongoclient.so without mozjs, the problem goes away (note lack of -lmozjs, and -L and -Wl,rpath arguments pointing to /usr/lib64/xulrunner-w.x.y.z:

mongo$ g++ -o libmongoclient.so.9.9.9 -fPIC -pthread -rdynamic -Wl,-soname,libmongoclient.so.9.9.9 -shared pch.os buildinfo.os db/common.os db/jsobj.os db/json.os db/lasterror.os db/nonce.os db/queryutil.os shell/mongo.os util/background.os util/mmap.os util/ramstore.os util/sock.os util/util.os util/message.os util/assert_util.os util/httpclient.os util/md5main.os util/base64.os util/concurrency/vars.os util/concurrency/task.os util/debug_util.os util/concurrency/thread_pool.os util/password.os util/version.os util/histogram.os util/concurrency/spin_lock.os util/text.os util/md5.os client/connpool.os client/dbclient.os client/dbclientcursor.os client/model.os client/syncclusterconnection.os s/shardconnection.os util/mmap_posix.os util/processinfo_linux2.os client/clientOnly.os client/gridfs.os s/d_util.os -L/usr/lib64 -L/lib64 -lpthread -lstdc++ -lboost_system-mt -lboost_thread-mt -lboost_filesystem-mt -lboost_program_options-mt -lpcrecpp -lpcre

mongo_libc_freeres$ ldd ./a.out
linux-vdso.so.1 => (0x00007fff3693c000)
libmongoclient.so.9.9.9 => ~/mongo.temp.install/lib64/libmongoclient.so.9.9.9 (0x00007f68fd175000)
libstdc+.so.6 => /usr/lib/libstdc+.so.6 (0x00007f68fce41000)
libm.so.6 => /lib/libm.so.6 (0x00007f68fcbbd000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f68fc9a6000)
libc.so.6 => /lib/libc.so.6 (0x00007f68fc624000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f68fc406000)
libboost_system.so.1.40.0 => /usr/lib/libboost_system.so.1.40.0 (0x00007f68fc202000)
libboost_thread.so.1.40.0 => /usr/lib/libboost_thread.so.1.40.0 (0x00007f68fbfec000)
libboost_filesystem.so.1.40.0 => /usr/lib/libboost_filesystem.so.1.40.0 (0x00007f68fbdd6000)
libboost_program_options.so.1.40.0 => /usr/lib/libboost_program_options.so.1.40.0 (0x00007f68fbb88000)
libpcrecpp.so.0 => /usr/lib/libpcrecpp.so.0 (0x00007f68fb97f000)
libpcre.so.3 => /lib/libpcre.so.3 (0x00007f68fb750000)
/lib64/ld-linux-x86-64.so.2 (0x00007f68fd64d000)
librt.so.1 => /lib/librt.so.1 (0x00007f68fb548000)

(note that there is no link to libmozjs listed above)

Re-running valgrind:

mongo_libc_freeres$ valgrind --track-origins=yes --read-var-info=yes --leak-check=full ./a.out
==29841== Memcheck, a memory error detector
==29841== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==29841== Using Valgrind-3.6.0.SVN and LibVEX; rerun with -h for copyright info
==29841== Command: ./a.out
==29841==
Hello, World
==29841==
==29841== HEAP SUMMARY:
==29841== in use at exit: 10,512 bytes in 25 blocks
==29841== total heap usage: 421 allocs, 396 frees, 40,988 bytes allocated
==29841==
==29841== LEAK SUMMARY:
==29841== definitely lost: 0 bytes in 0 blocks
==29841== indirectly lost: 0 bytes in 0 blocks
==29841== possibly lost: 0 bytes in 0 blocks
==29841== still reachable: 10,512 bytes in 25 blocks
==29841== suppressed: 0 bytes in 0 blocks
==29841== Reachable blocks (those to which a pointer was found) are not shown.
==29841== To see them, rerun with: --leak-check=full --show-reachable=yes
==29841==
==29841== For counts of detected and suppressed errors, rerun with: -v
==29841== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)

And our error goes away. Its pretty clear that libmongoclient doesn't actually depend on mozjs.so (since it linked), and forcing all clients that link against libmongoclient to pick up such a dependency seems to cause memory errors. It would be good if libmongoclient would drop this dependency.

FWIW, libmongoclient links, on my system, the following libraries:

-lpthread -lstdc++ -lboost_system-mt -lboost_thread-mt -lboost_filesystem-mt -lboost_program_options-mt -lpcrecpp -lpcre

Many of these are not necessary:

The explicit -lpthread is not needed since the g++ driver is being called with the -pthread option. Its better to let the compiler driver worry about where to put things like -lpthread and -lc on the link line.
The -lstdc++ is not necessary since that is included by default by the g++ driver when linking
The -lboost_program_options_mt library is not needed since the client library does not do any options parsing
The library does not appear to depend on -lpcrecpp -lpcre either.

I verified that all of these can be removed by manually linking the library with -Wl,-zdefs added to the link line and removing each of the libraries, one by one. The final link line looked like:

g++ -o libmongoclient.so.9.9.9 -fPIC -pthread -rdynamic -Wl,-zdefs -Wl,-soname,libmongoclient.so.9.9.9 -shared pch.os buildinfo.os db/common.os db/jsobj.os db/json.os db/lasterror.os db/nonce.os db/queryutil.os shell/mongo.os util/background.os util/mmap.os util/ramstore.os util/sock.os util/util.os util/message.os util/assert_util.os util/httpclient.os util/md5main.os util/base64.os util/concurrency/vars.os util/concurrency/task.os util/debug_util.os util/concurrency/thread_pool.os util/password.os util/version.os util/histogram.os util/concurrency/spin_lock.os util/text.os util/md5.os client/connpool.os client/dbclient.os client/dbclientcursor.os client/model.os client/syncclusterconnection.os s/shardconnection.os util/mmap_posix.os util/processinfo_linux2.os db/commands.os client/clientOnly.os client/gridfs.os s/d_util.os -L/usr/lib64 -L/lib64 -lboost_system-mt -lboost_thread-mt -lboost_filesystem-mt

After this, my test mainline still runs cleanly, even with LD_BIND_NOW in the environment:

mongo_libc_freeres$ LD_BIND_NOW=1 ./a.out
Hello, World

And still shows no errors under valgrind.



 Comments   
Comment by Andrew Morrow (Inactive) [ 27/Jul/10 ]

Yep, working great now. Thanks!

Comment by Michael Dirolf [ 27/Jul/10 ]

there was a bug in the scons options handling where that invocation was not picking up the --full option. this should be fixed now, can you try again?

thanks for the reports!

Comment by auto [ 27/Jul/10 ]

Author:

{'login': 'mdirolf', 'name': 'Mike Dirolf', 'email': 'mike@10gen.com'}

Message: minor: use store_true for --sharedclient SERVER-1266
http://github.com/mongodb/mongo/commit/eeb86f5ab90e9f8ae3e85b6ab7cf7b715bead021

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

OK, I just tried:

$ scons -c . ; scons -j 8 --prefix /home/acm/tmp/mongo-tmp-install --sharedclient --full install

It does build libmongoclient.so in my current source dir, but neither it nor the headers make it to the prefix directory:

$ cd /home/acm/tmp/mongo-tmp-install/
$ find . -type f
./bin/mongo
./bin/mongofiles
./bin/mongorestore
./bin/mongoimport
./bin/mongod
./bin/mongoexport
./bin/mongodump
./bin/mongos
./bin/mongosniff
./bin/mongostat

Can I send you some build log or are there SCons flags that would get debugging output that would be useful to you?

I'm currently building from:

http://github.com/mongodb/mongo/commit/127558c1c47f8df7fddd6a7408036eccdd3bf9ad

Thanks. (And sorry for all the noise on this)

Comment by Michael Dirolf [ 26/Jul/10 ]

You should still be able to build w/o building the separate client directory first (although that should work too).

There was an issue with --full that I just fixed, so can you try again? Should be something like:

scons --prefix whatever --sharedclient --full install

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

RE: --as-needed: awesome. Thanks.

RE building and packaging: I understand that its hard to get the build system working for everyone. But I'm confused about what I'm supposed to do to get a build that includes both libmongoclient.so and the BSON headers from the current git tree.

Do I need to do a two stage build, where I first build something with '--prefix mongo-cxx-driver', and then build the driver from the resulting mongo-cxx-driver directory? Or should I still be able to build libmongoclient.so and the BSON headers directly from the tree?

Because I've tried all sorts of variations of

scons --prefix <something> --sharedclient --full <install|mongoclient>

and in no case do I get both a shared libmongoclient.so and the client headers copied to the prefix directory, despite being built in the source tree.

Comment by auto [ 26/Jul/10 ]

Author:

{'login': 'erh', 'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: SERVER-1266 -Wl,as-needed flag for shared library
http://github.com/mongodb/mongo/commit/a85b659bb6cc722a407e0fea6c39d2a523382a78

Comment by Eliot Horowitz (Inactive) [ 26/Jul/10 ]

The problem comes down to supporting a build system for an infinite number of use cases.
The client has more places it needs to work - so splitting it out makes it easier for people.

I added Wl,-as-needed -Wl,-zdefs for both builds.

At some point we'll need to do some build system cleaning - but its hard given the complexity.

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

Hmm. That is unfortunate. It is unlikely that we are going to switch to using your client tarball, since we already build our own debian packages from your main source tree, onto which we rebase our changes for http://jira.mongodb.org/browse/SERVER-1257.

I guess I don't really understand why creating a separate client tarball is a better solution than fixing the build system: we really want to be able to build mongo, in all facets, from one source tree, then package the build artifacts into separate server, client, and -dev debian packages.

Anyway, I guess we can also maintain our preferred fix for this issue, which is to use Wl,-as-needed -Wl,-zdefs on the link line for libmongoclient.so, along with our debian packaging changes.

Comment by Michael Dirolf [ 26/Jul/10 ]

Working on a fix for this now - thanks!

Just as an FYI - the .so you get from a `scons --full --sharedclient install` will still link in libmozjs - to get around that you'll need to use the separate client tarball, see the case that eliot linked you to above.

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

OK:

'scons --full install' does install the headers, libs, and binaries, but gets me a libmongoclient.a
'scons --full install --sharedclient' gets me only a 'bin' directory, no headers, and notably, no library at all.

So I can't seem to get both a shared library, and the headers + libs.

Any ideas?

Comment by Michael Dirolf [ 26/Jul/10 ]

should be able to do
scons --full install

http://github.com/mongodb/mongo/commit/b7358a8ea958a14c052cb5ca4ded2a6bcc8cb0e5

Comment by Eliot Horowitz (Inactive) [ 26/Jul/10 ]

We're going to add an option for a full/driver install in the main one.

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

Ah. Is that why I suddenly can't get the mongo build system to install the libmongoclient header files unless I call my 'prefix' directory mongo-cxx-client?

http://github.com/mongodb/mongo/blob/master/SConstruct#L494

That seems a bit restrictive.

Comment by Eliot Horowitz (Inactive) [ 26/Jul/10 ]

We're going to package a separate c++ .tgz to keep things a bit cleaner for people who just want to use bson/c++ driver.
First version should be up soon
SEE: SERVER-1355

Comment by Andrew Morrow (Inactive) [ 26/Jul/10 ]

I just tried to build client libraries against http://github.com/mongodb/mongo/commit/3083c08000747ec21ed5d723fa444ae698b4108a as follows:

scons --prefix ~/tmp/mongo-tmp-install --release --nostrip --sharedclient mongoclient

which is failing as follows:

scons: Reading SConscript files ...
scons version: 1.2.0.d20100117
python version: 2 6 5 'final' 0
Checking whether the C++ compiler worksyes
Checking for C library stdc++... yes
Checking for C++ header file pcrecpp.h... yes
Checking for C++ header file boost/filesystem/operations.hpp... yes
Checking for C++ header file execinfo.h... yes
ERROR: can't find static version of: ['mozjs', 'js', 'js_static'] in: ['/usr/lib64/xulrunner-1.9.2.7', '/usr/lib64', '/lib64', '/usr/lib', '/usr/local/lib']

I'm not sure why it would be looking for static versions of mozjs when building the shared client libraries.

Also, I'm not sure what the 'new C++ driver thing / separate client tarball' comments are referring to.

Thanks.

Comment by Michael Dirolf [ 26/Jul/10 ]

no longer links in libmozjs w/ the separate client tarball.

Comment by Eliot Horowitz (Inactive) [ 24/Jul/10 ]

With the new c++ driver thing - this should be automatic - but this is just a sanity check.
resolve when you confirm.

Comment by Andrew Morrow (Inactive) [ 01/Jul/10 ]

So, obviously, this is really only a problem when building libmongoclient as a shared library. However, building it as a shared object is what we want to do, so:

We looked into updating the sconscript so that libmongoclient didn't link libmozjs.so, but that turned out to be rather difficult given the architecture of the sconscript. As a workaround, at least when the linker is a binutils ld (bfd or gold), one solution is to add 'Wl,-as-needed' to the link line. With this flag, link artifacts will only record DT_NEEDED entries for libraries from which a symbol was actually resolved at link time. So, since libmongoclient.so doesn't actually have any dependencies on libmozjs.so, it doesn't end up with a DT_NEEDED entry even though -lmozjs is on the link line, and libmozjs.so therefore doesn't get pulled into the address space of processes which link libmongoclient.so.

I'd recommend though, if building with Wl,-as-needed, to also build with -Wl,-zdefs, which is just about always the right thing to do when building a shared library.

So my recommended fix for this is to enable Wl,-as-needed -Wl,-zdefs on the link line if libmongoclient is to be built as a shared object on a platform using GNU ld.

Generated at Thu Feb 08 02:56:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.