[SERVER-5487] Seg fault when shutting down replica set member - Subscription Build with SSL Enabled Created: 03/Apr/12  Updated: 11/Jul/16  Resolved: 05/Dec/12

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 2.0.4
Fix Version/s: 2.2.3, 2.3.2

Type: Bug Priority: Minor - P4
Reporter: Adam Comerford Assignee: Eric Milkie
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Amazon Linux amzn-ami-pv-2012.03.1.x86_64-ebs; mongodb 2.0.4; git version: 329f3c47fe8136c03392c8f0e548506cb21f8ebf modules: subscription


Issue Links:
Depends
Duplicate
is duplicated by SERVER-8833 Segmentation fault after replSet reli... Closed
is duplicated by SERVER-7766 mongod with SSL segfaults when closin... Closed
is duplicated by SERVER-8643 Calling rs.remove() causes a segmenta... Closed
Related
related to SERVER-7766 mongod with SSL segfaults when closin... Closed
Operating System: ALL
Participants:

 Description   

Using this subscription build (2.0.4) for Amazon Linux:

http://downloads.10gen.com/linux/mongodb-linux-x86_64-subscription-amzn64-2.0.4.tgz

Shutting down a replica set instance (using SSL) using a SIGTERM (kill -15 <pid>) or a SIGINT (Ctrl-C), mongod does not exit cleanly



 Comments   
Comment by auto [ 08/Jan/13 ]

Author:

{u'date': u'2012-11-29T18:56:25Z', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-5487 do not call SSL_shutdown on an SSL context while another thread is using it
Branch: v2.2
https://github.com/mongodb/mongo/commit/8885784cf186b48bda5bd0ee99ff0bfa0a76ff69

Comment by auto [ 29/Nov/12 ]

Author:

{u'date': u'2012-11-29T18:56:25Z', u'email': u'milkie@10gen.com', u'name': u'Eric Milkie'}

Message: SERVER-5487 do not call SSL_shutdown on an SSL context while another thread is using it
Branch: master
https://github.com/mongodb/mongo/commit/8e7cbf47244f1fa83624ca67c4aada66da783198

Comment by Andy Schwerin [ 18/May/12 ]

One possible bandaid would be to not run the socket-closing thread on shutdown.

Closing sockets that belong to other, live threads is probably bad news, anyways. A more complete solution should come as part of a startup/shutdown redesign.

Comment by Andy Schwerin [ 18/May/12 ]

Reproduces on Ubuntu 12.04 in mongo 2.1.2-pre- (master branch). Changing ::exit() to ::_exit() does not change this behavior.

Comment by Adam Comerford [ 05/Apr/12 ]

Just to note, initial testing suggests this does not happen with the Ubuntu build, it seems to exit cleanly when run in a similar manner.

Comment by Adam Comerford [ 03/Apr/12 ]

Note: this does not happen if SSL is not enabled, without SSL the build exits cleanly.

To reproduce, provision an Amazon AWS instance (micro is just about big enough):

//install updates, install required libs
sudo yum update
sudo yum install net-snmp-libs

Get the subscription build, 2.0.4, and set up a replica set:

// make sure the data folder exists and has the right permissions and that the PEM file details are correct
./mongod --dbpath /data/db/r1 --sslOnNormalPorts --sslPEMKeyFile ~/test.pem --sslPEMKeyPassword "mongo" --replSet killtest --oplogSize 100 --noprealloc
./mongod --dbpath /data/db/r2 --sslOnNormalPorts --sslPEMKeyFile ~/test.pem --sslPEMKeyPassword "mongo" --replSet killtest --oplogSize 100 --noprealloc --port 27018

Initiate the replica set. Then either send the signal to the PID or hit Ctrl-C on the command line. Error looks like this:

Tue Apr  3 15:55:25 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends
Tue Apr  3 15:55:25 [interruptThread] now exiting
Tue Apr  3 15:55:25 dbexit: 
Tue Apr  3 15:55:25 [interruptThread] shutdown: going to close listening sockets...
Tue Apr  3 15:55:25 [interruptThread] closing listening socket: 6
Tue Apr  3 15:55:25 [interruptThread] closing listening socket: 7
Tue Apr  3 15:55:25 [interruptThread] closing listening socket: 8
Tue Apr  3 15:55:25 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Tue Apr  3 15:55:25 [interruptThread] shutdown: going to flush diaglog...
Tue Apr  3 15:55:25 [interruptThread] shutdown: going to close sockets...
Tue Apr  3 15:55:25 [interruptThread] shutdown: waiting for fs preallocator...
Tue Apr  3 15:55:25 [interruptThread] shutdown: lock for final commit...
Tue Apr  3 15:55:25 [interruptThread] shutdown: final commit...
Tue Apr  3 15:55:25 Invalid access at address: 0x7f6b74000178
 
Tue Apr  3 15:55:25 Got signal: 11 (Segmentation fault).
 
Tue Apr  3 15:55:25 Backtrace:
0x94fd52 0x950373 0x7f6b9bb834a0 0x7f6b74000178 
 ./mongod(_ZN5mongo10abruptQuitEi+0x3d2) [0x94fd52]
 ./mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x203) [0x950373]
 /lib64/libpthread.so.0(+0xf4a0) [0x7f6b9bb834a0]
 [0x7f6b74000178]
 
Logstream::get called in uninitialized state
Tue Apr  3 15:55:25 ERROR: Client::shutdown not called: conn

Generated at Thu Feb 08 03:09:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.