[SERVER-3825] Segmentation fault on CentOS 5.6 Created: 13/Sep/11  Updated: 30/Mar/12  Resolved: 13/Sep/11

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 2.0.0
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Stefan Huber Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: crash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 5.6 64bit


Attachments: File mongod-jeeves.strace     File mongod.conf    
Operating System: Linux
Participants:

 Description   

Made a clean install of mongodb 2.0.0 (e.g. removed all database files of an existing 1.8.3 installation prior try running the server). However the server instantly dies instantaneously

Tue Sep 13 01:04:42 BackgroundJob starting: DataFileSync
Tue Sep 13 01:04:42 isInRangeTest passed
Tue Sep 13 01:04:42 shardKeyTest passed
Tue Sep 13 01:04:42 shardObjTest passed
Tue Sep 13 01:04:42 versionCmpTest passed
Tue Sep 13 01:04:42 versionArrayTest passed
Tue Sep 13 01:04:42 [initandlisten] MongoDB starting : pid=30432 port=27017 dbpath=/var/mongodb 64-bit host=Jeeves
Tue Sep 13 01:04:42 [initandlisten] db version v2.0.0, pdfile version 4.5
Tue Sep 13 01:04:42 [initandlisten] git version: nogitversion
Tue Sep 13 01:04:42 [initandlisten] build info: Linux Jeeves 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 BOOST_LIB_VERSION=1_33_1
Tue Sep 13 01:04:42 [initandlisten] options:

{ bind_ip: "127.0.0.1", config: "/var/mongodb/mongod.conf", dbpath: "/var/mongodb", fork: "true", logappend: "true", logpath: "/var/log/mongodb/mongod.log", vvvv: true }

Tue Sep 13 01:04:42 [initandlisten] flushing directory /var/mongodb
Tue Sep 13 01:04:42 [initandlisten] journal dir=/var/mongodb/journal
Tue Sep 13 01:04:42 [initandlisten] recover : no journal files present, no recovery needed
Tue Sep 13 01:04:42 Invalid access at address: 0x10

Tue Sep 13 01:04:42 Got signal: 11 (Segmentation fault).

Tue Sep 13 01:04:42 Backtrace:
0xa58bf9 0xa591d0 0x3265a0eb10 0x72c971 0x738ebe 0x727baa 0xa5c8c9 0xa5d38a 0xa6546f 0x326461d994 0x4db589
mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa58bf9]
mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa591d0]
/lib64/libpthread.so.0 [0x3265a0eb10]
mongod(_ZN5mongo9writelockD1Ev+0x151) [0x72c971]
mongod(_ZN5mongo3dur7recoverEv+0x8e) [0x738ebe]
mongod(_ZN5mongo3dur7startupEv+0x2a) [0x727baa]
mongod(_ZN5mongo14_initAndListenEi+0x3d9) [0xa5c8c9]
mongod(_ZN5mongo13initAndListenEi+0x2a) [0xa5d38a]
mongod(main+0x79cf) [0xa6546f]
/lib64/libc.so.6(__libc_start_main+0xf4) [0x326461d994]
mongod(__gxx_personality_v0+0x459) [0x4db589]

Logstream::get called in uninitialized state
Tue Sep 13 01:04:42 ERROR: Client::shutdown not called: initandlisten

I've attached the strace of running
'strace -s 100 mongod --port 9999 --dbpath /tmp/does_not_exist 2>mongod.strace'

An 1.8.3 installation on the same server runs without any problems using the same configuration.



 Comments   
Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

great

Comment by Stefan Huber [ 13/Sep/11 ]

just rebuild using a newer version of boost (namely 1.41)
Works as expected now.

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

That version of boost is known to have bugs, can you try updating?

Comment by Stefan Huber [ 13/Sep/11 ]

BOOST_LIB_VERSION=1_33_1

Building is essentially done using these steps:
cd /opt/src && wget http://downloads.mongodb.org/src/mongodb-src-r2.0.0.tar.gz
tar xzf mongodb-src-r2.0.0.tar.gz && cd mongodb-src-r2.0.0
scons --prefix=/opt/mongodb-2.0.0 install

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

what version of boost do you have installed, and how did you compile?

Comment by Stefan Huber [ 13/Sep/11 ]

The binary was build directly on the server the same way as various 1.8 builds before.
Just tried the official binary which works flawlessly.

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

Where is this binary from?
Can you try with an official binary?

Comment by Stefan Huber [ 13/Sep/11 ]

I created the directory just for testing e.g. it was completely empty before running mongodb.
Afterwards:

[root@Jeeves ~]# ls -la /data/db
total 16
drwxr-xr-x 3 root root 4096 Sep 13 18:33 .
drwxr-xr-x 3 root root 4096 Sep 13 18:33 ..
drwxr-xr-x 2 root root 4096 Sep 13 18:33 journal
-rwxr-xr-x 1 root root 6 Sep 13 18:33 mongod.lock

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

What's in /data/db/

Comment by Stefan Huber [ 13/Sep/11 ]

Still the same issue using an empty directory and without my mongod.conf file

[root@Jeeves ~]# /opt/mongodb-2.0.0/bin/mongod -vvv
Tue Sep 13 18:33:55 BackgroundJob starting: DataFileSync
Tue Sep 13 18:33:55 isInRangeTest passed
Tue Sep 13 18:33:55 shardKeyTest passed
Tue Sep 13 18:33:55 shardObjTest passed
Tue Sep 13 18:33:55 versionCmpTest passed
Tue Sep 13 18:33:55 versionArrayTest passed
Tue Sep 13 18:33:55 [initandlisten] MongoDB starting : pid=11906 port=27017 dbpath=/data/db/ 64-bit host=Jeeves
Tue Sep 13 18:33:55 [initandlisten] db version v2.0.0, pdfile version 4.5
Tue Sep 13 18:33:55 [initandlisten] git version: nogitversion
Tue Sep 13 18:33:55 [initandlisten] build info: Linux Jeeves 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 BOOST_LIB_VERSION=1_33_1
Tue Sep 13 18:33:55 [initandlisten] options:

{ vvv: true }

Tue Sep 13 18:33:55 [initandlisten] flushing directory /data/db
Tue Sep 13 18:33:55 [initandlisten] journal dir=/data/db/journal
Tue Sep 13 18:33:55 [initandlisten] recover : no journal files present, no recovery needed
Tue Sep 13 18:33:55 Invalid access at address: 0x10

Tue Sep 13 18:33:55 Got signal: 11 (Segmentation fault).

Tue Sep 13 18:33:55 Backtrace:
0xa58bf9 0xa591d0 0x3265a0eb10 0x72c971 0x738ebe 0x727baa 0xa5c8c9 0xa5d38a 0xa6546f 0x326461d994 0x4db589
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa58bf9]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa591d0]
/lib64/libpthread.so.0 [0x3265a0eb10]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo9writelockD1Ev+0x151) [0x72c971]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo3dur7recoverEv+0x8e) [0x738ebe]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo3dur7startupEv+0x2a) [0x727baa]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo14_initAndListenEi+0x3d9) [0xa5c8c9]
/opt/mongodb-2.0.0/bin/mongod(_ZN5mongo13initAndListenEi+0x2a) [0xa5d38a]
/opt/mongodb-2.0.0/bin/mongod(main+0x79cf) [0xa6546f]
/lib64/libc.so.6(__libc_start_main+0xf4) [0x326461d994]
/opt/mongodb-2.0.0/bin/mongod(__gxx_personality_v0+0x459) [0x4db589]

Logstream::get called in uninitialized state
Tue Sep 13 18:33:55 ERROR: Client::shutdown not called: initandlisten

Edited: Include right log message

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

Can you remove the old journal directory?

Comment by Stefan Huber [ 13/Sep/11 ]

[root@Jeeves ~]# ls -la /var/mongodb/
total 20
drwxr-xr-x 3 mongod mongod 4096 Sep 13 01:18 .
drwxr-xr-x 21 root root 4096 Jul 18 01:53 ..
drwxr-xr-x 2 mongod mongod 4096 Sep 13 01:20 journal
rw-rr- 1 root root 1635 Sep 13 01:16 mongod.conf
-rwxr-xr-x 1 mongod mongod 6 Sep 13 01:18 mongod.lock
[root@Jeeves ~]#

I attached my config file which I forgot in the first place.

Comment by Eliot Horowitz (Inactive) [ 13/Sep/11 ]

The 2.0 starts seems to be using /var/mongodb
Is there anything in there?

Generated at Thu Feb 08 03:04:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.