[SERVER-43116] mongodb3.6.6 sharding noed crashed Created: 31/Aug/19  Updated: 01/Oct/19  Resolved: 01/Oct/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: dan li Assignee: Danny Hatcher (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

2019-08-29T10:25:53.752+0800 I NETWORK [conn140455] end connection 172.16.50.182:62823 (19 connections now open)
2019-08-29T10:26:26.143+0800 I NETWORK [listener] connection accepted from 172.16.50.103:39594 #140456 (20 connections now open)
2019-08-29T10:26:26.165+0800 I NETWORK [conn140456] received client metadata from 172.16.50.103:39594 conn140456: \{ driver: { name: "PyMongo", version: "3.7.2" }, os: \{ type: "Linux", name: "CentOS Linux 7.4.1708 Core", architecture: "x86_64", version: "3.10.0-693.el7.x86_64" }, platform: "CPython 2.7.5.final.0" }
2019-08-29T10:26:26.198+0800 I NETWORK [listener] connection accepted from 172.16.50.103:39630 #140457 (21 connections now open)
2019-08-29T10:26:26.198+0800 I NETWORK [conn140457] received client metadata from 172.16.50.103:39630 conn140457: \{ driver: { name: "PyMongo", version: "3.7.2" }, os: \{ type: "Linux", name: "CentOS Linux 7.4.1708 Core", architecture: "x86_64", version: "3.10.0-693.el7.x86_64" }, platform: "CPython 2.7.5.final.0" }
2019-08-29T10:26:26.307+0800 I ACCESS [conn140457] SCRAM-SHA-1 authentication failed for lepus_monitor on admin from client 172.16.50.103:39630 ; AuthenticationFailed: SCRAM-SHA-1 authentication failed, storedKey mismatch
2019-08-29T10:26:26.418+0800 I NETWORK [conn140456] end connection 172.16.50.103:39594 (20 connections now open)
2019-08-29T10:26:26.418+0800 I NETWORK [conn140457] end connection 172.16.50.103:39630 (19 connections now open)
2019-08-29T10:26:50.651+0800 F - [conn55839] Invariant failure opCtx->lockState()->isDbLockedForMode(db, MODE_IS) src/mongo/db/catalog/database_holder_impl.cpp 100
2019-08-29T10:26:50.657+0800 F - [conn55839]
 
***aborting after invariant() failure
 
 
2019-08-29T10:26:50.783+0800 F - [conn55839] Got signal: 6 (Aborted).
 
0x559cf1c05aa1 0x559cf1c04cb9 0x559cf1c0519d 0x7f9218ac67e0 0x7f92187554f5 0x7f9218756cd5 0x559cf034573c 0x559cf06d8878 0x559cf09b1936 0x559cf09b1bb5 0x559cf08b37a2 0x559cf0721885 0x559cf072199e 0x559cf0abb648 0x559cf03d9a49 0x559cf1539826 0x559cf153444f 0x559cf05d53ac 0x559cf05d730d 0x559cf05d8194 0x559cf05e48ea 0x559cf05e0297 0x559cf05e36d1 0x559cf14f4c02 0x559cf05df0ff 0x559cf05e1645 0x559cf05e1f3b 0x559cf05e031d 0x559cf05e36d1 0x559cf14f5165 0x559cf1ab97b4 0x7f9218abeaa1 0x7f921880bc4d
----- BEGIN BACKTRACE -----
{"backtrace":[\{"b":"559CEF9B2000","o":"2253AA1","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"559CEF9B2000","o":"2252CB9"},\{"b":"559CEF9B2000","o":"225319D"},\{"b":"7F9218AB7000","o":"F7E0"},\{"b":"7F9218723000","o":"324F5","s":"gsignal"},\{"b":"7F9218723000","o":"33CD5","s":"abort"},\{"b":"559CEF9B2000","o":"99373C","s":"_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j"},\{"b":"559CEF9B2000","o":"D26878","s":"_ZNK5mongo18DatabaseHolderImpl3getEPNS_16OperationContextENS_10StringDataE"},\{"b":"559CEF9B2000","o":"FFF936","s":"_ZN5mongo16OldClientContext11_finishInitEv"},\{"b":"559CEF9B2000","o":"FFFBB5","s":"_ZN5mongo16OldClientContextC2EPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb"},\{"b":"559CEF9B2000","o":"F017A2","s":"_ZN5mongo7Helpers12putSingletonEPNS_16OperationContextEPKcNS_7BSONObjE"},\{"b":"559CEF9B2000","o":"D6F885"},\{"b":"559CEF9B2000","o":"D6F99E","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl26storeLocalLastVoteDocumentEPNS_16OperationContextERKNS0_8LastVoteE"},\{"b":"559CEF9B2000","o":"1109648","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl26processReplSetRequestVotesEPNS_16OperationContextERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseE"},\{"b":"559CEF9B2000","o":"A27A49","s":"_ZN5mongo4repl22CmdReplSetRequestVotes3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},\{"b":"559CEF9B2000","o":"1B87826","s":"_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},\{"b":"559CEF9B2000","o":"1B8244F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},\{"b":"559CEF9B2000","o":"C233AC"},\{"b":"559CEF9B2000","o":"C2530D"},\{"b":"559CEF9B2000","o":"C26194","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},\{"b":"559CEF9B2000","o":"C328EA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},\{"b":"559CEF9B2000","o":"C2E297","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"559CEF9B2000","o":"C316D1"},\{"b":"559CEF9B2000","o":"1B42C02","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},\{"b":"559CEF9B2000","o":"C2D0FF","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},\{"b":"559CEF9B2000","o":"C2F645","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},\{"b":"559CEF9B2000","o":"C2FF3B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},\{"b":"559CEF9B2000","o":"C2E31D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},\{"b":"559CEF9B2000","o":"C316D1"},\{"b":"559CEF9B2000","o":"1B43165"},\{"b":"559CEF9B2000","o":"21077B4"},\{"b":"7F9218AB7000","o":"7AA1"},\{"b":"7F9218723000","o":"E8C4D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "3.6.6", "gitVersion" : "6405d65b1d6432e138b44c13085d0c2fe235d6bd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.el6.x86_64", "version" : "#1 SMP Tue Jun 19 21:26:04 UTC 2018", "machine" : "x86_64" }, "somap" : [ \{ "b" : "559CEF9B2000", "elfType" : 3, "buildId" : "7D732317C59BE71FD914B0A82F65C004C8826D7A" }, \{ "b" : "7FFC3216B000", "elfType" : 3, "buildId" : "A527E0E44603433B61EA716072C7A30506AD5000" }, \{ "b" : "7F9219BCB000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4786A2A5D30B121601958E84D643C70C13C4FBA5" }, \{ "b" : "7F92197E6000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "CBDA444A7109874C5350AE9CEEF3F82F749B347F" }, \{ "b" : "7F921957A000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BECFB85A8BC084042D5BF2BA9E66325CE798B659" }, \{ "b" : "7F9219376000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2AF795BFFD122309BA3359FEBABB5D0967403D17" }, \{ "b" : "7F921916E000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "552CEC3216281CCFD7FA6432C723D50163255823" }, \{ "b" : "7F9218EEA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4AAEE970B045D8BF946578B9C7F3AB5CDE9AB44A" }, \{ "b" : "7F9218CD4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, \{ "b" : "7F9218AB7000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4EA475CD3FD3B69B6C95D9381FA74B36DB4992EF" }, \{ "b" : "7F9218723000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "BCA7789C2EA8E28CB7CE553E183AC7E7EE36F8A2" }, \{ "b" : "7F9219DE5000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, \{ "b" : "7F921850D000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7F92182C9000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, \{ "b" : "7F9217FE2000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, \{ "b" : "7F9217DDE000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, \{ "b" : "7F9217BB2000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, \{ "b" : "7F92179A7000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, \{ "b" : "7F92177A4000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7F9217585000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x559cf1c05aa1]
 mongod(+0x2252CB9) [0x559cf1c04cb9]
 mongod(+0x225319D) [0x559cf1c0519d]
 libpthread.so.0(+0xF7E0) [0x7f9218ac67e0]
 libc.so.6(gsignal+0x35) [0x7f92187554f5]
 libc.so.6(abort+0x175) [0x7f9218756cd5]
 mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x559cf034573c]
 mongod(_ZNK5mongo18DatabaseHolderImpl3getEPNS_16OperationContextENS_10StringDataE+0x188) [0x559cf06d8878]
 mongod(_ZN5mongo16OldClientContext11_finishInitEv+0x46) [0x559cf09b1936]
 mongod(_ZN5mongo16OldClientContextC2EPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEb+0x55) [0x559cf09b1bb5]
 mongod(_ZN5mongo7Helpers12putSingletonEPNS_16OperationContextEPKcNS_7BSONObjE+0x82) [0x559cf08b37a2]
 mongod(+0xD6F885) [0x559cf0721885]
 mongod(_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl26storeLocalLastVoteDocumentEPNS_16OperationContextERKNS0_8LastVoteE+0x8E) [0x559cf072199e]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl26processReplSetRequestVotesEPNS_16OperationContextERKNS0_23ReplSetRequestVotesArgsEPNS0_27ReplSetRequestVotesResponseE+0x178) [0x559cf0abb648]
 mongod(_ZN5mongo4repl22CmdReplSetRequestVotes3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0x1F9) [0x559cf03d9a49]
 mongod(_ZN5mongo12BasicCommand11enhancedRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x76) [0x559cf1539826]
 mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x559cf153444f]
 mongod(+0xC233AC) [0x559cf05d53ac]
 mongod(+0xC2530D) [0x559cf05d730d]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B4) [0x559cf05d8194]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x559cf05e48ea]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x559cf05e0297]
 mongod(+0xC316D1) [0x559cf05e36d1]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x559cf14f4c02]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x559cf05df0ff]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x559cf05e1645]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x559cf05e1f3b]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x559cf05e031d]
 mongod(+0xC316D1) [0x559cf05e36d1]
 mongod(+0x1B43165) [0x559cf14f5165]
 mongod(+0x21077B4) [0x559cf1ab97b4]
 libpthread.so.0(+0x7AA1) [0x7f9218abeaa1]
 libc.so.6(clone+0x6D) [0x7f921880bc4d]
----- END BACKTRACE -----



 Comments   
Comment by Danny Hatcher (Inactive) [ 03/Sep/19 ]

Could you please try upgrading your cluster to 3.6.14 and see if the problem happens again? In SERVER-42055 we changed the way that code path was accessed which should avoid the invariant issue.

Generated at Thu Feb 08 05:02:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.