|
See SERVER-2929
|
|
log from one entire run and next run
|
|
it took some time to crash
|
|
entire log
|
|
Is this right after startup? Can send the entire mongos log?
|
|
ok. this bug will be for the crash
|
|
Looks like something happened at 10:37 eastern time.
Any ideas what? Anything in logs?
Are you aggregrating logs?
You should open a new case if you want help investigating that so this case can be about the crash itself.
|
|
and another crash
Received signal 11
Backtrace: Fri Apr 8 13:25:00 [conn15501] end connection 127.0.0.1:52647
Fri Apr 8 13:25:00 [conn15449] end connection 127.0.0.1:51508
0x52e235 0x2b126b013070 0x6213b3 0x69afab 0x576ba6 0x5774b6 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x43)[0x6213b3]
/usr/bin/mongos[0x69afab]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x576ba6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x5774b6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
Fri Apr 8 13:25:00 [conn15480] end connection 127.0.0.1:51853
Fri Apr 8 13:25:00 [conn15508] end connection 127.0.0.1:52664
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
Received signal 11
Backtrace: Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54414 #15588
Received signal 11
Backtrace: Received signal 11
Backtrace: Received signal 11
Backtrace: Received signal 11
Backtrace: 0x52e235 0x2b126b013070 0x5317e0 0x575514 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x575514]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
0x52e235 0x2b126b013070 0x5317e0 0x575514 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x575514]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
0x52e235 0x2b126b013070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
0x52e235 0x2b126b013070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54416 #15589
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
Received signal 11
Backtrace: /usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54418 #15590
0x52e235 0x2b126b013070 0x4e59cc 0x4ed03e 0x4ed7f6 0x4ee484 0x618864 0x635783 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2b126a63c1b5 0x2b126b0b036d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2b126b013070]
/usr/bin/mongos(_ZN5mongo10FieldRangeC1ERKNS_11BSONElementEbb+0x18c)[0x4e59cc]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo13FieldRangeSet17processQueryFieldERKNS_11BSONElementEb+0x5e)[0x4ed03e]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2b126a63c1b5]
Fri Apr 8 13:25:00 [mongosMain] connection accepted from 127.0.0.1:54419 #15591
/lib64/libc.so.6(clone+0x6d)[0x2b126b0b036d]
===
|
|
Hi,
my system is running very bad now
looks like requests takes too much time.
can you please look at the mms to see if you see something wrong ?
nothing interesting in the logs
thanks,
ofer
|
|
now it crashed on 1.8.1
Fri Apr 8 09:53:14 [mongosMain] connection accepted from 127.0.0.1:56506 #10447
Fri Apr 8 09:53:14 [mongosMain] connection accepted from 127.0.0.1:56507 #10448
Received signal 11
Backtrace: 0x52e235 0x2af336b5c070 0x6213b3 0x61539e 0x69ac16 0x576ba6 0x5774b6 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2
af336bf936d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x43)[0x6213b3]
/usr/bin/mongos(_ZN5mongo15setShardVersionERNS_12DBClientBaseERKSsNS_17ShardChunkVersionEbRNS_7BSONObjE+0x44e)[0x61539e]
/usr/bin/mongos[0x69ac16]
/usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi+0x16)[0x576ba6]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6)[0x5774b6]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
Received signal 11
Backtrace: 0x52e235 0x2af336b5c070 0x5317e0 0x533243 0x57750d 0x575630 0x575b31 0x651627 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2af336bf936d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x575b31]
/usr/bin/mongos(_ZN5mongo15dbgrid_pub_cmds16FindAndModifyCmd3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x3c7)[0x651627]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
===
^@336bf936d ^@11
===
^@336bf936d ^@11
Backtrace: 0x52e235 0x2af336b5c070 0x5317e0 0x533243 0x57750d 0x575630 0x575886 0x583914 0x586464 0x66c41a 0x6362c9 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x2af3361851b5 0x2af336bf93
6d
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x2af336b5c070]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x5317e0]
/usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x43)[0x533243]
/usr/bin/mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x21d)[0x57750d]
/usr/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0)[0x575630]
/usr/bin/mongos(ZN5mongo15ShardConnectionC1ERKSsS2+0x76)[0x575886]
/usr/bin/mongos(_ZN5mongo15ClusteredCursor5queryERKSsiNS_7BSONObjEi+0x124)[0x583914]
/usr/bin/mongos(_ZN5mongo27SerialServerClusteredCursor4moreEv+0x134)[0x586464]
/usr/bin/mongos(_ZN5mongo19ShardedClientCursor13sendNextBatchERNS_7RequestEi+0x8a)[0x66c41a]
/usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xe39)[0x6362c9]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/usr/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x2af3361851b5]
/lib64/libc.so.6(clone+0x6d)[0x2af336bf936d]
===
===
^@336bf936d ^@11
===
^@336bf936d ^@11
===
^@336bf936d ^@11
Fri Apr 8 09:58:52 /usr/bin/mongos db version v1.8.1, pdfile version 4.5 starting (--help for usage)
Fri Apr 8 09:58:52 git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04
|
|
mongos has successfully upgraded
mongo hosts will be upgrade next week
thanks,\ofer
|
|
Yes - its best to keep everything in sync and there some fixes in mongod
|
|
Hi,
I'm going to upgrade all mongos today to 1.8.1
should I upgrade mongo hosts and arbs as well ?
thanks,
ofer
|
|
waiting for 1.8.1
thanks,
ofer
|
|
it should be truly random, but with enough mongos, can see things like that.
have you upgraded?
|
|
Hi,
any reason why this crash occured on 4 mongos last 48 hours ?
this crash is new to me.
thanks,
ofer
|
|
This should be fixed in 1.8.1-rc1
You can either try that now, or wait for 1.8.1 early next week (approx. 4/6)
|
Generated at Thu Feb 08 03:01:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.