[SERVER-3082] mongos crash on shutdown Created: 11/May/11  Updated: 10/Dec/14  Resolved: 02/May/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 1.7.3, 1.8.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Allen Gilbert Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 1
Labels: crash, mongos, neweng, sharding, shutdown
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

EC2 m2.xlarge instance running Amazon linux 2.6.34.7-56.40.amzn1.x86_64


Issue Links:
Depends
Operating System: Linux
Participants:

 Description   

Whenever I attempt to shut down mongos via db.shutdownServer() after running it with active clients, shutdown fails with the following error:

Wed May 11 09:18:58 /opt/mongodb/bin/mongos db version v1.8.1, pdfile version 4.5 starting (--help for usage)
Wed May 11 09:18:58 git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04
Wed May 11 09:18:58 build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
...
Wed May 11 09:26:25 [conn2] terminating, shutdown command received
Wed May 11 09:26:25 [conn2] dbexit: shutdown called rc:0 shutdown called
Wed May 11 09:26:25 [mongosMain] dbexit: rc:0
Received signal 11
Backtrace: 0x52e235 0x3fb5832730 0x3fb5835c35 0x6700a9 0x5b2819 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(exit+0x35)[0x3fb5835c35]
/opt/mongodb/bin/mongos[0x6700a9]
/opt/mongodb/bin/mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x89)[0x5b2819]
/opt/mongodb/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/opt/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Received signal 6
Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
/opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===

Additionally, this entry appears in /var/log/messages:
May 11 09:26:25 ip-10-243-3-246 mongos: *** glibc detected *** /opt/mongodb/bin/mongos: free(): invalid pointer: 0x0000000000ac9020 ***

Shutdown also fails when using the "kill -2 PID" approach:

Wed May 11 13:17:39 [mongosMain] dbexit: received signal 2 rc:0 received signal 2
Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
Received signal 6
Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
/opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===



 Comments   
Comment by Greg Studer [ 02/May/14 ]

Resolving since this issue has become stale - feel free to reopen if the issue persists in newer mongos versions.

Comment by Pierre Baillet [ 15/Jun/12 ]

The bug still exists in 2.0.2 :

mongo.log

2012-06-15_08:23:02.09506 *** glibc detected *** /opt/mongodb/bin/mongos: free(): invalid pointer: 0x0000000000a71920 ***
2012-06-15_08:23:02.13443 ======= Backtrace: =========
2012-06-15_08:23:02.13447 /lib/libc.so.6(+0x774b6)[0x7f5af35c14b6]
2012-06-15_08:23:02.13447 /lib/libc.so.6(cfree+0x73)[0x7f5af35c7c83]
2012-06-15_08:23:02.15092 /opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x167)[0x50c077]
2012-06-15_08:23:02.15095 /opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x79)[0x50d9e9]
2012-06-15_08:23:02.15257 /opt/mongodb/bin/mongos(thread_proxy+0x6e)[0x76ebee]
2012-06-15_08:23:02.15259 /lib/libpthread.so.0(+0x7971)[0x7f5af4073971]
2012-06-15_08:23:02.15260 /lib/libc.so.6(clone+0x6d)[0x7f5af363092d]

Note than when we see the issue, it's upon restarting mongo that behaves curiously on the client level:

[08:15:45.703857] #<Mongo::OperationFailure: dbclient error communicating with server: m13:27101>
[08:15:45.704699] /ftn/apps/picor/production/shared/bundle/ruby/1.8/gems/mongo-1.3.0/lib/../lib/mongo/cursor.rb:89:in `next_document'
[08:15:45.705397] /ftn/apps/picor/production/shared/bundle/ruby/1.8/gems/mongo-1.3.0/lib/../lib/mongo/cursor.rb:236:in `each'
[08:15:45.706137] /ftn/apps/picor/production/shared/bundle/ruby/1.8/gems/mongo-1.3.0/lib/../lib/mongo/db.rb:222:in `collect'
[08:15:45.706878] /ftn/apps/picor/production/shared/bundle/ruby/1.8/gems/mongo-1.3.0/lib/../lib/mongo/db.rb:222:in `collection_names'
[08:15:45.707641] /mnt/apps/picor/production/releases/14777fc6f0f7cfaec348094d7c86a2f035d47df2/lib/mongo_temporary.rb:19:in `gc_old_temporaries'
[08:15:45.708388] /mnt/apps/picor/production/releases/14777fc6f0f7cfaec348094d7c86a2f035d47df2/lib/mongo_temporary.rb:17:in `each'
[08:15:45.709464] /mnt/apps/picor/production/releases/14777fc6f0f7cfaec348094d7c86a2f035d47df2/lib/mongo_temporary.rb:17:in `gc_old_temporaries'
[08:15:45.710418] /mnt/apps/picor/production/releases/14777fc6f0f7cfaec348094d7c86a2f035d47df2/lib/mongo_temporary.rb:15:in `each'
[08:15:45.711121] /mnt/apps/picor/production/releases/14777fc6f0f7cfaec348094d7c86a2f035d47df2/lib/mongo_temporary.rb:15:in `gc_old_temporaries'

Generated at Thu Feb 08 03:02:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.