[SERVER-37565] mongod continuously restarting after upgrade to 4.0.2 Created: 11/Oct/18  Updated: 29/Oct/23  Resolved: 20/Dec/18

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: 4.0.2, 4.0.3
Fix Version/s: 4.0.6, 4.1.7

Type: Bug Priority: Major - P3
Reporter: Andrada Nastasie Assignee: Patrick Freed
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive diagnostic.data.zip    
Issue Links:
Backports
Problem/Incident
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:
  • start with mongod version 3.6.7
  • stop mongod
  • upgrade to 4.0.1/4.0.2/4.0.3
  • start mongod
  • 600-700 connections to the upgraded member in the first 2 minutes
  • server restarts

This is a trace from the mongodb log, maybe it helps:

{"backtrace":[{"b":"7FF94AF13000","o":"23B8EA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FF94AF13000","o":"23B80B9"},{"b":"7FF94AF13000","o":"23B859D"},{"b":"7FF94966C000","o":"11390"},{"b":"7FF9492A2000","o":"35428","s":"gsignal"},{"b":"7FF9492A2000","o":"3702A","s":"abort"},{"b":"7FF94AF13000","o":"A04503","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7FF94AF13000","o":"234D657","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"7FF94AF13000","o":"1DA8D9D","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE"},{"b":"7FF94AF13000","o":"1DA9BF8","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE"},{"b":"7FF94AF13000","o":"1B51B40"},{"b":"7FF94AF13000","o":"1B537C0"},{"b":"7FF94AF13000","o":"1E2A3F9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"7FF94AF13000","o":"AAE3DF"},{"b":"7FF94AF13000","o":"AB060F"},{"b":"7FF94AF13000","o":"AB2979"},{"b":"7FF94AF13000","o":"AB38B1","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"7FF94AF13000","o":"AA0EFA","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"7FF94AF13000","o":"AABC6A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"7FF94AF13000","o":"AA6937","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7FF94AF13000","o":"AAA141"},{"b":"7FF94AF13000","o":"1C5D902","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"7FF94AF13000","o":"AA4B4F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"7FF94AF13000","o":"AA7CE5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"7FF94AF13000","o":"AA6077","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"7FF94AF13000","o":"AA69BD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7FF94AF13000","o":"AAA141"},{"b":"7FF94AF13000","o":"1C5DE65"},{"b":"7FF94AF13000","o":"2312724"},{"b":"7FF94966C000","o":"76BA"},{"b":"7FF9492A2000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.2", "gitVersion" : "fc1573ba18aee42f97a3bb13b67af7d837826b47", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-107-generic", "version" : "#154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "7FF94AF13000", "elfType" : 3, "buildId" : "AAF90A0B71749BBE7B9E3CF14359342D6DCFF400" }, { "b" : "7FFEABCBC000", "elfType" : 3, "buildId" : "C1463BCC3775232AE965C05F22AD5EDF8D26BEA5" }, { "b" : "7FF94AA7C000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5D192460BE804CC9B9807E6D49834431CBF20E9A" }, { "b" : "7FF94A861000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7FF94A41D000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7FF94A1B4000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7FF949FB0000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7FF949DA8000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7FF949A9F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7FF949889000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FF94966C000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7FF9492A2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7FF94ACEB000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }, { "b" : "7FF94906F000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7FF948E53000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7FF948C09000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7FF9489FA000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "D3B183C41F02C7CD18F906AAFD19C69C850F1CEB" }, { "b" : "7FF9487A9000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "DA0EC53E16B3AC6BDC56EAEFE1BFECDDC395FB2E" }, { "b" : "7FF94858F000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7FF94825F000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "3CE893F6D1382C2C7648DCCB06E71B1C7E0861CC" }, { "b" : "7FF94802C000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FF947DF6000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FF947B76000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FF9478A4000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7FF947675000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7FF947471000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7FF947266000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7FF94704B000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7FF946E0A000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7FF946BA6000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FF946993000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7FF94678F000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FF946586000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7FF9462FC000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7FF94605A000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7FF945E27000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7FF945C11000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7FF945A09000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7FF9457E0000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType&...
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7ff94d2cbea1]
 mongod(+0x23B80B9) [0x7ff94d2cb0b9]
 mongod(+0x23B859D) [0x7ff94d2cb59d]
 libpthread.so.0(+0x11390) [0x7ff94967d390]
 libc.so.6(gsignal+0x38) [0x7ff9492d7428]
 libc.so.6(abort+0x16A) [0x7ff9492d902a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7ff94b917503]
 mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x737) [0x7ff94d260657]
 mongod(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE+0xBDD) [0x7ff94ccbbd9d]
 mongod(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE+0x98) [0x7ff94ccbcbf8]
 mongod(+0x1B51B40) [0x7ff94ca64b40]
 mongod(+0x1B537C0) [0x7ff94ca667c0]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x7ff94cd3d3f9]
 mongod(+0xAAE3DF) [0x7ff94b9c13df]
 mongod(+0xAB060F) [0x7ff94b9c360f]
 mongod(+0xAB2979) [0x7ff94b9c5979]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3C1) [0x7ff94b9c68b1]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x7ff94b9b3efa]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x7ff94b9bec6a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x7ff94b9b9937]
 mongod(+0xAAA141) [0x7ff94b9bd141]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x7ff94cb70902]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x7ff94b9b7b4f]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x7ff94b9bace5]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x7ff94b9b9077]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x7ff94b9b99bd]
 mongod(+0xAAA141) [0x7ff94b9bd141]
 mongod(+0x1C5DE65) [0x7ff94cb70e65]
 mongod(+0x2312724) [0x7ff94d225724]
 libpthread.so.0(+0x76BA) [0x7ff9496736ba]
 libc.so.6(clone+0x6D) [0x7ff9493a941d]
-----  END BACKTRACE  -----

Sprint: Security 2018-12-03, Security 2018-12-17, Security 2018-12-31
Participants:
Case:

 Description   

Hello,

I am trying to upgrade a replicaset from version 3.6.7 to 4.0.2. I started with one of the secondaries, everything was ok in the first 2 minutes after the upgrade but then the server keeps restarting every 2 or 3 minutes. All the members of this replicaset have a lot of connections (~1000) but after the server restarts it gets ~600-700 connections and then it crashes again. 

I have upgraded a different replicaset which doesn't have this many connections and it worked fine so I think this is the issue here.



 Comments   
Comment by Githook User [ 22/Jan/19 ]

Author:

{'email': 'patrick.freed@mongodb.com', 'name': 'Patrick Freed', 'username': 'patrickfreed'}

Message: SERVER-37565 Unlock memory held during SCRAM authentication

This fixes a bug where the server would crash if a large number of parallel connections occurred at once

(cherry picked from commit 916a5553a2db8ae7553fea7c3703ef8fef75b055)
Branch: v4.0
https://github.com/mongodb/mongo/commit/5a176fcf616ca6e461205058d359f3137e486773

Comment by Githook User [ 20/Dec/18 ]

Author:

{'username': 'patrickfreed', 'email': 'patrick.freed@mongodb.com', 'name': 'Patrick Freed'}

Message: SERVER-37565 Unlock memory held during SCRAM authentication

This fixes a bug where the server would crash if a large number of parallel connections occurred at once
Branch: master
https://github.com/mongodb/mongo/commit/916a5553a2db8ae7553fea7c3703ef8fef75b055

Comment by Kelsey Schubert [ 14/Nov/18 ]

Hi andrada,

Glad to hear the issue was resolved by reconfiguring your ulimits. There were some changes to how secure memory is allocated during authentication in 4.0, which caused you to stumble into this issue. We're still discussing the best approach to remedy this behavior, and we'll continue using this ticket to to track this issue until we determine the next steps to improve performance in this space.

Thanks,
Kelsey

Comment by Andrada Nastasie [ 24/Oct/18 ]

Hi Kelsey,

Thank you very much for the recommendation, I did not have the memlock ulimit set appropriately. After I set it correctly, the upgrade worked very well. But tell me, is there any change in mongo 4, because mongo 3.6 and prior worked very well with the default ulimits (64 kb).

Thank you,
Andrada

Comment by Kelsey Schubert [ 19/Oct/18 ]

Hi andrada,

Would you please ensure that your ulimits are set appropriately? In particular, I'd like to confirm that your memlock is unlimited. Please review https://docs.mongodb.com/manual/reference/ulimit/index.html#unix-ulimit-settings for additional details regarding these configurations.

Thank you,
Kelsey

Comment by Andrada Nastasie [ 19/Oct/18 ]

Hello,

Do you have any news regarding my question?

Thank you,
Andrada

Comment by Andrada Nastasie [ 12/Oct/18 ]

Hello again,

I wanted to let you know that I just upgraded another replicaset with ~200 connections per member and everything went just fine.

Thank you,
Andrada

Comment by Andrada Nastasie [ 12/Oct/18 ]

Hello Dan and Ramon,

Thank you for your quick answers. I attached you the diagnostic.data and here is a bigger chunk of the log:

2018-10-12T08:16:43.782+0000 I NETWORK  [conn5243] received client metadata from 172.24.167.135:37488 conn5243: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.9.0-5-amd64" }, platform: "Node.js v9.5.0, LE, mongodb-core: 2.1.20" }
2018-10-12T08:16:43.782+0000 I NETWORK  [conn5238] received client metadata from 172.24.167.219:60544 conn5238: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.9.0-5-amd64" }, platform: "Node.js v10.11.0, LE, mongodb-core: 2.1.20" }
2018-10-12T08:16:43.782+0000 I NETWORK  [conn4729] end connection 172.24.177.91:45478 (419 connections now open)
2018-10-12T08:16:43.784+0000 F -        [conn5011] Got signal: 6 (Aborted).
 0x7f4425737ea1 0x7f44257370b9 0x7f442573759d 0x7f4421ae9390 0x7f4421743428 0x7f442174502a 0x7f4423d83503 0x7f44256cc657 0x7f4425127d9d 0x7f4425128bf8 0x7f4424ed0b40 0x7f4424ed27c0 0x7f44251a93f9 0x7f4423e2d3df 0x7f4423e2f60f 0x7f4423e31979 0x7f4423e328b1 0x7f4423e1fefa 0x7f4423e2ac6a 0x7f4423e25937 0x7f4423e29141 0x7f4424fdc902 0x7f4423e23b4f 0x7f4423e26ce5 0x7f4423e25077 0x7f4423e259bd 0x7f4423e29141 0x7f4424fdce65 0x7f4425691724 0x7f4421adf6ba 0x7f442181541d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F442337F000","o":"23B8EA1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F442337F000","o":"23B80B9"},{"b":"7F442337F000","o":"23B859D"},{"b":"7F4421AD8000","o":"11390"},{"b":"7F442170E000","o":"35428","s":"gsignal"},{"b":"7F442170E000","o":"3702A","s":"abort"},{"b":"7F442337F000","o":"A04503","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7F442337F000","o":"234D657","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"7F442337F000","o":"1DA8D9D","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE"},{"b":"7F442337F000","o":"1DA9BF8","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE"},{"b":"7F442337F000","o":"1B51B40"},{"b":"7F442337F000","o":"1B537C0"},{"b":"7F442337F000","o":"1E2A3F9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"7F442337F000","o":"AAE3DF"},{"b":"7F442337F000","o":"AB060F"},{"b":"7F442337F000","o":"AB2979"},{"b":"7F442337F000","o":"AB38B1","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"7F442337F000","o":"AA0EFA","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"7F442337F000","o":"AABC6A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"7F442337F000","o":"AA6937","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7F442337F000","o":"AAA141"},{"b":"7F442337F000","o":"1C5D902","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"7F442337F000","o":"AA4B4F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"7F442337F000","o":"AA7CE5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"7F442337F000","o":"AA6077","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"7F442337F000","o":"AA69BD","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7F442337F000","o":"AAA141"},{"b":"7F442337F000","o":"1C5DE65"},{"b":"7F442337F000","o":"2312724"},{"b":"7F4421AD8000","o":"76BA"},{"b":"7F442170E000","o":"10741D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.2", "gitVersion" : "fc1573ba18aee42f97a3bb13b67af7d837826b47", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-107-generic", "version" : "#154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "7F442337F000", "elfType" : 3, "buildId" : "AAF90A0B71749BBE7B9E3CF14359342D6DCFF400" }, { "b" : "7FFC486FE000", "elfType" : 3, "buildId" : "C1463BCC3775232AE965C05F22AD5EDF8D26BEA5" }, { "b" : "7F4422EE8000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5D192460BE804CC9B9807E6D49834431CBF20E9A" }, { "b" : "7F4422CCD000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7F4422889000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7F4422620000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7F442241C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F4422214000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F4421F0B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F4421CF5000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F4421AD8000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F442170E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F4423157000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }, { "b" : "7F44214DB000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "E09D3783AD1D0BBCD3204FA01E4EF6D756E18F57" }, { "b" : "7F44212BF000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7F4421075000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7F4420E66000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "D3B183C41F02C7CD18F906AAFD19C69C850F1CEB" }, { "b" : "7F4420C15000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "DA0EC53E16B3AC6BDC56EAEFE1BFECDDC395FB2E" }, { "b" : "7F44209FB000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7F44206CB000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "3CE893F6D1382C2C7648DCCB06E71B1C7E0861CC" }, { "b" : "7F4420498000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7F4420262000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7F441FFE2000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7F441FD10000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7F441FAE1000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7F441F8DD000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7F441F6D2000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7F441F4B7000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7F441F276000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7F441F012000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7F441EDFF000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "E07E186694852D8F69459C6AB28A53F8DA3CE3B6" }, { "b" : "7F441EBFB000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7F441E9F2000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7F441E768000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7F441E4C6000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7F441E293000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7F441E07D000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7F441DE75000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7F441DC4C000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7F441DA3D000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7F441D7F2000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7F441D51D000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7F441D2E5000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7BDD51353D50310FFA1587E4AA01B40ABE32D582" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f4425737ea1]
 mongod(+0x23B80B9) [0x7f44257370b9]
 mongod(+0x23B859D) [0x7f442573759d]
 libpthread.so.0(+0x11390) [0x7f4421ae9390]
 libc.so.6(gsignal+0x38) [0x7f4421743428]
 libc.so.6(abort+0x16A) [0x7f442174502a]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7f4423d83503]
 mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x737) [0x7f44256cc657]
 mongod(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE+0xBDD) [0x7f4425127d9d]
 mongod(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE+0x98) [0x7f4425128bf8]
 mongod(+0x1B51B40) [0x7f4424ed0b40]
 mongod(+0x1B537C0) [0x7f4424ed27c0]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x7f44251a93f9]
 mongod(+0xAAE3DF) [0x7f4423e2d3df]
 mongod(+0xAB060F) [0x7f4423e2f60f]
 mongod(+0xAB2979) [0x7f4423e31979]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x3C1) [0x7f4423e328b1]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3A) [0x7f4423e1fefa]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x7f4423e2ac6a]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x7f4423e25937]
 mongod(+0xAAA141) [0x7f4423e29141]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x7f4424fdc902]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x7f4423e23b4f]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x7f4423e26ce5]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x7f4423e25077]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x7f4423e259bd]
 mongod(+0xAAA141) [0x7f4423e29141]
 mongod(+0x1C5DE65) [0x7f4424fdce65]
 mongod(+0x2312724) [0x7f4425691724]
 libpthread.so.0(+0x76BA) [0x7f4421adf6ba]
 libc.so.6(clone+0x6D) [0x7f442181541d]
-----  END BACKTRACE  -----
2018-10-12T08:16:44.445+0000 I CONTROL  [main] ***** SERVER RESTARTED *****

Comment by Ramon Fernandez Marina [ 11/Oct/18 ]

andrada, as Dan points out we'll need the full logs: the backtrace above is incomplete and I was unable to symbolize it.

If you can also please upload the contents of the diagnostic.data directory for one of the upgraded members that may provide additional information.

Thanks,
Ramón.

Comment by Daniel Pasette (Inactive) [ 11/Oct/18 ]

can you attach the complete log or at least include the lines above the backtrace?

Generated at Thu Feb 08 04:46:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.