[SERVER-2889] mongos crashed Created: 02/Apr/11  Updated: 12/Jul/16  Resolved: 11/Apr/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.8.0
Fix Version/s: 1.9.0

Type: Bug Priority: Major - P3
Reporter: ofer samocha Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongos.tar.gz    
Operating System: Linux
Participants:

 Description   

mongos crashed

Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32935 #339189
Sat Apr 2 06:20:16 [conn339168] end connection 127.0.0.1:32915
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32936 #339190
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32937 #339191
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32938 #339192
Sat Apr 2 06:20:16 [conn339141] end connection 127.0.0.1:32887
Sat Apr 2 06:20:16 [conn339121] end connection 127.0.0.1:32867
Sat Apr 2 06:20:16 [conn339173] end connection 127.0.0.1:32920
Sat Apr 2 06:20:16 [conn339156] end connection 127.0.0.1:32903
Sat Apr 2 06:20:16 [conn339131] end connection 127.0.0.1:32879
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32939 #339193
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32940 #339194
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32941 #339195
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32942 #339196
Sat Apr 2 06:20:16 [conn339072] end connection 127.0.0.1:32818
Sat Apr 2 06:20:16 [conn339152] end connection 127.0.0.1:32899
Sat Apr 2 06:20:16 [conn339179] end connection 127.0.0.1:32926
Sat Apr 2 06:20:16 [conn339175] end connection 127.0.0.1:32922
Received signal 11
Backtrace: Sat Apr 2 06:20:16 [conn339176] end connection 127.0.0.1:32923
Sat Apr 2 06:20:16 [conn339166] end connection 127.0.0.1:32913
Sat Apr 2 06:20:16 [conn339185] end connection 127.0.0.1:32932
Sat Apr 2 06:20:16 [conn339178] end connection 127.0.0.1:32925
Sat Apr 2 06:20:16 [conn339186] end connection 127.0.0.1:32933
0x5319a5 0x2b47e452d070 0x2b47e3e06ac4 0x624906 0x69e8ca 0x5799e6 0x57a2f6 0x578470 0x5786c6 0x586644 0x589194 0x66fa0a 0x639585 0x667d6c 0x67ae11 0x5817a2 0x6a22b0 0x2b47e3b561b5 0x2b47e45ca36d Sat Apr 2 06:20:16 [conn339154] end connection 127.0.0.1:32901
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x5319a5]
/lib64/libc.so.6[0x2b47e452d070]
/usr/local/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x24)[0x2b47e3e06ac4]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x86)[0x624906]
/usr/bin/mongos[0x69e8ca]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x5799e6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x57a2f6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x578470]
/usr/bin/mongos(ZN5mongo15ShardConnectionC1ERKSsS2+0x76)[0x5786c6]
/usr/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0x124)[0x586644]
/usr/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134)[0x589194]
/usr/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a)[0x66fa0a]
/usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe55)[0x639585]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x667d6c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x141)[0x67ae11]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x272)[0x5817a2]
/usr/bin/mongos(thread_proxy+0x80)[0x6a22b0]
/usr/bin/mongos(thread_proxy+0x80)[0x6a22b0]
/lib64/libpthread.so.0[0x2b47e3b561b5]
/lib64/libc.so.6(clone+0x6d)[0x2b47e45ca36d]
===
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32943 #339197
Received signal 11
Backtrace: 0x5319a5 0x2b47e452d070 0x534f50 0x5369b3 0x57a34d 0x578470 0x578971 0x654c39 0x57ec0c 0x634572 0x667d6c 0x67ae11 0x5817a2 0x6a22b0 0x2b47e3b561b5 0x2b47e45ca36d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x5319a5]
/lib64/libc.so.6[0x2b47e452d070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x534f50]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x5369b3]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57a34d]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x578470]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x578971]
Sat Apr 2 06:20:16 [mongosMain] connection accepted from 127.0.0.1:32944 #339198
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3b9)[0x654c39]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57ec0c]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x634572]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x667d6c]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x634572]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x667d6c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x141)[0x67ae11]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x272)[0x5817a2]
/usr/bin/mongos(thread_proxy+0x80)[0x6a22b0]
/lib64/libpthread.so.0[0x2b47e3b561b5]
/lib64/libc.so.6(clone+0x6d)[0x2b47e45ca36d]
===
Received signal 11
Backtrace: 0x5319a5 0x2b47e452d070 0x5fb694 0x66706d 0x67adfb 0x5817a2 0x6a22b0 0x2b47e3b561b5 0x2b47e45ca36d
Sat Apr 2 06:20:16 [conn339196] Assertion failure _name.size() s/shard.h 75
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x5319a5]
/lib64/libc.so.6[0x2b47e452d070]
/usr/bin/mongos(_ZN5mongo4Grid11getDBConfigESsbRKSs+0x254)[0x5fb694]
/usr/bin/mongos(_ZN5mongo7Request5resetEb+0xbd)[0x66706d]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x12b)[0x67adfb]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x272)[0x5817a2]
/usr/bin/mongos(thread_proxy+0x80)[0x6a22b0]
/lib64/libpthread.so.0[0x2b47e3b561b5]
/lib64/libc.so.6(clone+0x6d)[0x2b47e45ca36d]
===
Received signal 6
Backtrace: 0x5319a5 0x2b47e452d070 0x2b47e452d015 0x2b47e452e980 0x2b47e45644cb 0x2b47e456f0d6 0x62607b 0x622967 0x2b47e452fd35 0x531a0b 0x2b47e452d070 0x2b47e3e06ac4 0x624906 0x69e8ca 0x5799e6 0x57a2f6 0x578470 0x5786c6 0x586644 0x589194
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x5319a5]
/lib64/libc.so.6[0x2b47e452d070]
/lib64/libc.so.6(gsignal+0x35)[0x2b47e452d015]
/lib64/libc.so.6(abort+0x110)[0x2b47e452e980]
/lib64/libc.so.6[0x2b47e45644cb]
/lib64/libc.so.6(cfree+0x1b6)[0x2b47e456f0d6]
/usr/bin/mongos(_ZNSt8_Rb_treeISsSt4pairIKSsN5mongo5ShardEESt10_Select1stIS4_ESt4lessISsESaIS4_EE8_M_eraseEPSt13_Rb_tree_nodeIS4_E+0x99b)[0x62607b]
/usr/bin/mongos[0x622967]
/lib64/libc.so.6(exit+0xe5)[0x2b47e452fd35]
/usr/bin/mongos[0x531a0b]
/lib64/libc.so.6[0x2b47e452d070]
/usr/local/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x24)[0x2b47e3e06ac4]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x86)[0x624906]
/usr/bin/mongos[0x69e8ca]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x5799e6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x57a2f6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x578470]
/usr/bin/mongos(ZN5mongo15ShardConnectionC1ERKSsS2+0x76)[0x5786c6]
/usr/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0x124)[0x586644]
/usr/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134)[0x589194]



 Comments   
Comment by Eliot Horowitz (Inactive) [ 11/Apr/11 ]

See SERVER-2929

Comment by ofer samocha [ 08/Apr/11 ]

log from one entire run and next run

Comment by ofer samocha [ 08/Apr/11 ]

it took some time to crash

Comment by ofer samocha [ 08/Apr/11 ]

entire log

Comment by Eliot Horowitz (Inactive) [ 08/Apr/11 ]

Is this right after startup? Can send the entire mongos log?

Comment by ofer samocha [ 08/Apr/11 ]

ok. this bug will be for the crash

Comment by Eliot Horowitz (Inactive) [ 08/Apr/11 ]

Looks like something happened at 10:37 eastern time.
Any ideas what? Anything in logs?
Are you aggregrating logs?
You should open a new case if you want help investigating that so this case can be about the crash itself.

Comment by ofer samocha [ 08/Apr/11 ]

and another crash

Received signal 11
Backtrace: Fri Apr 8 13:25:00 [conn15501] end connection 127.0.0.1:52647
Fri Apr 8 13:25:00 [conn15449] end connection 127.0.0.1:51508
0x52e235 0x2b126b013070 0x6213b3 0x69afab 0x576ba6 0x5774b6 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x43)[0x6213b3]
/usr/bin/mongos[0x69afab]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x576ba6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x5774b6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
Fri Apr 8 13:25:00 [conn15480] end connection 127.0.0.1:51853
Fri Apr 8 13:25:00 [conn15508] end connection 127.0.0.1:52664
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
Received signal 11
Backtrace: Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54414 #15588
Received signal 11
Backtrace: Received signal 11
Backtrace: Received signal 11
Backtrace: Received signal 11
Backtrace: 0x52e235 0x2b126b013070 0x5317e0 0x575514 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x575514]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
0x52e235 0x2b126b013070 0x5317e0 0x575514 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x575514]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
0x52e235 0x2b126b013070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
0x52e235 0x2b126b013070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54416 #15589
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54418 #15590
0x52e235 0x2b126b013070 0x4e59cc 0x4ed03e 0x4ed7f6 0x4ee484 0x618864 0x635783 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbb+0x18c)[0x4e59cc]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x5e)[0x4ed03e]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54419 #15591
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===

Comment by ofer samocha [ 08/Apr/11 ]

Hi,

my system is running very bad now
looks like requests takes too much time.
can you please look at the mms to see if you see something wrong ?
nothing interesting in the logs

thanks,
ofer

Comment by ofer samocha [ 08/Apr/11 ]

now it crashed on 1.8.1

Fri Apr 8 09:53:14 [mongosMain] connection accepted from 127.0.0.1:56506 #10447
Fri Apr 8 09:53:14 [mongosMain] connection accepted from 127.0.0.1:56507 #10448
Received signal 11
Backtrace: 0x52e235 0x2af336b5c070 0x6213b3 0x61539e 0x69ac16 0x576ba6 0x5774b6 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2
af336bf936d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x43)[0x6213b3]
/usr/bin/mongos(_ZN5mongo15setShardVersionERNS_12DBClientBaseERKSsNS_17ShardChunkVersionEbRNS_7BSONObjE+0x44e)[0x61539e]
/usr/bin/mongos[0x69ac16]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x576ba6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x5774b6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
Received signal 11
Backtrace: 0x52e235 0x2af336b5c070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2af336bf936d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
===
^@336bf936d ^@11
===
^@336bf936d ^@11
Backtrace: 0x52e235 0x2af336b5c070 0x5317e0 0x533243 0x57750d 0x575630 0x575886 0x583914 0x586464 0x66c41a 0x6362c9 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2af336bf93
6d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(ZN5mongo15ShardConnectionC1ERKSsS2+0x76)[0x575886]
/usr/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0x124)[0x583914]
/usr/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134)[0x586464]
/usr/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a)[0x66c41a]
/usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe39)[0x6362c9]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
===
^@336bf936d ^@11
===
^@336bf936d ^@11
===
^@336bf936d ^@11
Fri Apr 8 09:58:52 /usr/bin/mongos db version v1.8.1, pdfile version 4.5 starting (--help for usage)
Fri Apr 8 09:58:52 git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04

Comment by ofer samocha [ 07/Apr/11 ]

mongos has successfully upgraded

mongo hosts will be upgrade next week

thanks,\ofer

Comment by Eliot Horowitz (Inactive) [ 07/Apr/11 ]

Yes - its best to keep everything in sync and there some fixes in mongod

Comment by ofer samocha [ 07/Apr/11 ]

Hi,

I'm going to upgrade all mongos today to 1.8.1
should I upgrade mongo hosts and arbs as well ?

thanks,
ofer

Comment by ofer samocha [ 04/Apr/11 ]

waiting for 1.8.1

thanks,
ofer

Comment by Eliot Horowitz (Inactive) [ 04/Apr/11 ]

it should be truly random, but with enough mongos, can see things like that.
have you upgraded?

Comment by ofer samocha [ 04/Apr/11 ]

Hi,

any reason why this crash occured on 4 mongos last 48 hours ?
this crash is new to me.

thanks,
ofer

Comment by Eliot Horowitz (Inactive) [ 03/Apr/11 ]

This should be fixed in 1.8.1-rc1
You can either try that now, or wait for 1.8.1 early next week (approx. 4/6)

Generated at Thu Feb 08 03:01:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.