[SERVER-7404] Solaris file relocation error: posix_fadvise Created: 18/Oct/12  Updated: 11/Jul/16  Resolved: 20/Jun/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 2.3.2
Fix Version/s: 2.4.6, 2.5.1

Type: Bug Priority: Major - P3
Reporter: Tatyana Knaifel Assignee: Tad Marshall
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Solaris 10


Issue Links:
Depends
is depended on by SERVER-9761 Mongo executables should be built wit... Closed
Related
related to DOCS-1518 Document additional hints for running... Closed
is related to SERVER-9325 Add blockCheckSupported() features to... Closed
Backwards Compatibility: Fully Compatible
Operating System: Solaris
Participants:

 Description   

Mongod crashes

it crashes with error in the log:
ld.so.1: mongod: fatal: relocation error: file /opt/mongodb-sunos5-x86_64-2.2.1-rc0/bin/mongod: symbol posix_fadvise: referenced symbol not found
version 2.2.0 - worked OK.

All the log is the following:

Tue Oct 16 09:23:26 [initandlisten] MongoDB starting : pid=29854 port=27017
dbpath=/var/lib/mongodb 64-bit host=zimus2-10
Tue Oct 16 09:23:26 [initandlisten]
Tue Oct 16 09:23:26 [initandlisten] ** NOTE: your operating system version
does not support the method that MongoDB
Tue Oct 16 09:23:26 [initandlisten] ** uses to detect impending page
faults.
Tue Oct 16 09:23:26 [initandlisten] ** This may result in slower
performance for certain use cases
Tue Oct 16 09:23:26 [initandlisten]
Tue Oct 16 09:23:26 [initandlisten] db version v2.2.1-rc0, pdfile version
4.5
Tue Oct 16 09:23:26 [initandlisten] git version:
c8dadfd8f735e01d4d1c7435ab634c889a431674
Tue Oct 16 09:23:26 [initandlisten] build info: SunOS
bs-smartos-x86-64-1.10gen.cc 5.11 joyent_20120424T232010Z i86pc
BOOST_LIB_VERSION=1_49
Tue Oct 16 09:23:26 [initandlisten] options: { bind_ip: "10.0.0.21", config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logpath: "/var/log/mongodb/mongod.log", noauth: "true", oplogSize: 4096, pidfilepath: "/tmp/mongodb.pid", replSet: "some_replset", rest: "true" }
Tue Oct 16 09:23:26 [initandlisten] journal dir=/var/lib/mongodb/journal
Tue Oct 16 09:23:26 [initandlisten] recover : no journal files present, no
recovery needed
ld.so.1: mongod: fatal: relocation error: file
/opt/mongodb-sunos5-x86_64-2.2.1-rc0/bin/mongod: symbol posix_fadvise:
referenced symbol not found



 Comments   
Comment by auto [ 18/Jul/13 ]

Author:

{u'username': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-7404 Link to posix_fadvise at runtime for Solaris

For the Solaris/SmartOS build, do not make direct calls to posix_fadvise,
which is present in Solaris 11 but not in Solaris 10. Instead, see if
it is available in a loaded library (which will be libc.so.1) at runtime
and either call it or call an emulation. The emulation is a no-op.
Branch: v2.4
https://github.com/mongodb/mongo/commit/4d4790146b76d8e65caa9fc9c34dc4263e0539a9

Comment by auto [ 19/Jun/13 ]

Author:

{u'username': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-7404 Visual Studio – add posix_fadvise.

{cpp,h}

Branch: master
https://github.com/mongodb/mongo/commit/714a9c6febdac4d062811cbe09477d662e890fb6

Comment by auto [ 19/Jun/13 ]

Author:

{u'username': u'tadmarshall', u'name': u'Tad Marshall', u'email': u'tad@10gen.com'}

Message: SERVER-7404 Link to posix_fadvise at runtime for Solaris

For the Solaris/SmartOS build, do not make direct calls to posix_fadvise,
which is present in Solaris 11 but not in Solaris 10. Instead, see if
it is available in a loaded library (which will be libc.so.1) at runtime
and either call it or call an emulation. The emulation is a no-op.
Branch: master
https://github.com/mongodb/mongo/commit/69d7632211e20d05f564e67e0cb9fd98929e5d5a

Comment by Tad Marshall [ 15/May/13 ]

Thanks Serge! I'll add that information to DOCS-1518.

Comment by Serge Fonville [ 15/May/13 ]

Hi Tad,

The system used runs OpenIndiana 151a7

$ uname -srvmpi
SunOS 5.11 oi_151a7 i86pc i386 i86pc

Comment by Tad Marshall [ 15/May/13 ]

Hi serge.fonville@gmail.com,

Thanks for the note above about using a package from OpenCSW to get MongoDB running.

Can you tell us which OS and version you used that was fixed with these steps? I filed DOCS-1518 to try to get more information about working around issues like this online, and it would be great to know exactly which OS versions benefit from which workarounds.

Thanks!

Tad

Comment by Tad Marshall [ 15/May/13 ]

Hi Dan,

Thanks for the update!

I haven't tested on OmniOS yet but I will give that attention as well as I test the fix for SERVER-9325.

Thanks for the note that the opencsw build of libstdc++ makes 2.4.3 work on OmniOS. I'll file a documentation ticket to get that information into our online info.

Tad

Comment by Dan Kimmel [ 15/May/13 ]

Hey Tad, this is OmniOS, which shares the kernel (illumos) with SmartOS but uses a different set of userland packages - the sources for the packages are probably very similar but they are built differently.

I had used the mongo-extra-64 files from http://downloads.mongodb.org.s3.amazonaws.com/sunos5/. Using the opencsw package instead works like a charm with 2.4.3 - I should have read Serge's post more carefully.

Also, thanks for the info on detecting impending page faults!

Comment by Tad Marshall [ 14/May/13 ]

Hi Dan,

Which OS version is this? Solaris 10 or 11 or SmartOS?

Did you download the mongo-extra-64.tgz file from http://downloads.mongodb.org.s3.amazonaws.com/sunos5/ or the opencsw version serge.fonville@gmail.com suggested above?

SERVER-9325 should address the message "NOTE: your operating system version does not support the method that MongoDB uses to detect impending page faults."

SERVER-8910 may be a Boost problem, but it happens for me in Solaris as well (until I add the "export LC_ALL=C" workaround).

It would be good to get to the bottom of the relocation errors, so knowing the exact OS version where they happen would be helpful, as well as knowing whether the OpenCSW library fixes it in all cases.

Thanks!

Tad

Comment by Dan Kimmel [ 14/May/13 ]

I ran into this issue in 2.4.3 as well:

[dk ~/mongodb-sunos5-x86_64-2.4.3]$ export LD_LIBRARY_PATH=mongo-extra-64 # to fix the issue with Solaris here: http://docs.mongodb.org/ecosystem/tutorial/joyent-cloud/
[dk ~/mongodb-sunos5-x86_64-2.4.3]$ export LC_ALL=C # to fix the bug here: https://jira.mongodb.org/browse/SERVER-8910
[dk ~/mongodb-sunos5-x86_64-2.4.3]$ bin/mongod --dbpath=/home/dan/db/
Tue May 14 03:35:59.684 [initandlisten] MongoDB starting : pid=5880 port=27017 dbpath=/home/dan/db/ 64-bit host=dk
Tue May 14 03:35:59.684 [initandlisten] 
Tue May 14 03:35:59.684 [initandlisten] ** NOTE: your operating system version does not support the method that MongoDB
Tue May 14 03:35:59.685 [initandlisten] **       uses to detect impending page faults.
Tue May 14 03:35:59.685 [initandlisten] **       This may result in slower performance for certain use cases
Tue May 14 03:35:59.685 [initandlisten] 
Tue May 14 03:35:59.685 [initandlisten] db version v2.4.3
Tue May 14 03:35:59.685 [initandlisten] git version: fe1743177a5ea03e91e0052fb5e2cb2945f6d95f
Tue May 14 03:35:59.685 [initandlisten] build info: SunOS bs-smartos-x86-64-1.10gen.cc 5.11 joyent_20130322T181205Z i86pc BOOST_LIB_VERSION=1_49
Tue May 14 03:35:59.685 [initandlisten] allocator: system
Tue May 14 03:35:59.685 [initandlisten] options: { dbpath: "/home/dan/db/" }
Tue May 14 03:35:59.692 [initandlisten] journal dir=/home/dan/db/journal
Tue May 14 03:35:59.710 [initandlisten] recover : no journal files present, no recovery needed
ld.so.1: mongod: fatal: relocation error: file bin/mongod: symbol _ZNSt8__detail15_List_node_base7_M_hookEPS0_: referenced symbol not found
Killed

It seems to be an issue for all releases going back to 2.2.0, which is the last 64-bit build that worked for me on illumos with the current mongo-extra-64 tarball. From elsewhere online, it seems like the symbol mismatch is just a matter of compiling mongo-extra-64 with a different compiler version than the program binaries (but I could be wrong).

Comment by Serge Fonville [ 30/Mar/13 ]

I had the exact same issue with 2.4.1 mongod.
After downloading and installing libstdc++6-4.7.2\,REV\=2013.03.28-SunOS5.10-i386-CSW.pkg from opencsw it starts as usual

what I did:

wget http://mirror.opencsw.org/opencsw/allpkgs/libstdc%2b%2b6-4.7.2%2cREV%3d2013.03.28-SunOS5.10-i386-CSW.pkg.gz
gzip -d libstdc++6-4.7.2\,REV\=2013.03.28-SunOS5.10-i386-CSW.pkg.gz
pkgadd -d libstdc++6-4.7.2\,REV\=2013.03.28-SunOS5.10-i386-CSW.pkg
ln -s /opt/csw/lib/amd64/libstdc++.so.6.0.17 in /usr/lib/amd64/libstdc++.so.6

Comment by Eric Milkie [ 26/Nov/12 ]

Inadvertently resolved.
The posix_fadvise issue is still outstanding.

Comment by Eric Milkie [ 15/Nov/12 ]

Ok so it's possible that your libstdc++ doesn't have the symbol that ours does – ours is 6.0.16 and your is 6.0.14. Since the symbol is internal to the library (we are not calling it directly), you can try building from source, or you can try upgrading to obtain a newer libstdc++.

Comment by Ivan [ 15/Nov/12 ]

Eric sorry,

ls -la ../../gcc/4.5/lib/amd64/libstdc++.so.6
lrwxrwxrwx   1 root     root          19 Aug  2 18:28 ../../gcc/4.5/lib/amd64/libstdc++.so.6 -> libstdc++.so.6.0.14

Comment by Eric Milkie [ 15/Nov/12 ]

Hi Ivan,
At this point, if you're willing to try compiling from source, I'd suggest doing that. Is the libstd++.so.6 file in gcc/4.5/lib/amd64/libstdc++.so.6 a symlink as well or the actual library file? I was hoping to see a minor version embedded in the final filename.

Comment by Ivan [ 15/Nov/12 ]

Hi Eric,

I'm using compiled mongodb from http://www.mongodb.org/downloads (Solaris 64-bit):

root@host:/opt/bin# ldd mongod
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libresolv.so.2 =>        /lib/64/libresolv.so.2
        libstdc++.so.6 =>        /usr/lib/64/libstdc++.so.6
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        libgcc_s.so.1 =>         /usr/lib/64/libgcc_s.so.1
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libmd.so.1 =>    /lib/64/libmd.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
root@host:/opt/bin# ls -la /usr/lib/64/libstdc++.so.6
lrwxrwxrwx   1 root     root          38 Aug  2 18:28 /usr/lib/64/libstdc++.so.6 -> ../../gcc/4.5/lib/amd64/libstdc++.so.6

MongoDB version 2.0.7 download at same url working fine.

To resolve this problem need compiling mongo from source?
Tnx for you reply.

Comment by Eric Milkie [ 15/Nov/12 ]

Hi Ivan,
That symbol in your error message is not posix_fadvise, so I think that's a separate issue.
The symbol comes from the libstdc++ library. When we build mongod, we use SmartOS and link to libstdc++.so.6.0.16.

Can you check what version of the library you have on your system? You can see with "ldd":

# ldd mongod
   <etc...>
    libstdc++.so.6 => /opt/local/lib/amd64/libstdc++.so.6

When I look at that file on our build system, it symlinks to libstdc++.so.6.0.16.

Comment by Ivan [ 15/Nov/12 ]

SunOS HOSTNAME 5.11 11.0 i86pc i386 i86pc:
crashes too:
ld.so.1: mongod: fatal: relocation error: file /opt/bin/mongod: symbol ZNSt8detail15_List_node_base7_M_hookEPS0: referenced symbol not found

Comment by Eric Milkie [ 18/Oct/12 ]

There may be other regressions in Solaris 10 besides this. We currently have no way of testing Oracle Solaris 10.

Comment by Eric Milkie [ 18/Oct/12 ]

Oracle Solaris 10 doesn't have this symbol, but it does exist on Oracle Solaris 11. It does also exist on SmartOS but is a no-op. Thus, Solaris 11 and SmartOS will not have this issue.

Generated at Thu Feb 08 03:14:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.