[SERVER-61005] rs.initiate() fails with "Invariant failure" under specific startup options Created: 26/Oct/21  Updated: 29/Oct/23  Resolved: 13/Dec/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.3
Fix Version/s: 5.3.0, 5.0.6

Type: Bug Priority: Major - P3
Reporter: Igor Solodovnikov Assignee: Huayu Ouyang
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.1, v5.0
Steps To Reproduce:

Run the following script to reproduce the issue:

rm -rf reprodir
mkdir reprodir
 
openssl rand -base64 756 > $PWD/reprodir/reprodir.key
chmod 400 $PWD/reprodir/reprodir.key
 
mongod  --bind_ip_all --auth --dbpath=$PWD/reprodir --port=27017 --replSet=rs0 --storageEngine=wiredTiger --tlsAllowInvalidCertificates --clusterAuthMode=keyFile --keyFile=$PWD/reprodir/reprodir.key --wiredTigerCacheSizeGB=0.25 --logpath $PWD/reprodir/mongod.log --fork --profile=1 --verbose
 
sleep 5
mongo --eval 'rs.initiate({"_id":"rs0","version":1,"members":[{"_id":0,"host":"127.0.0.1:27017"}]})' admin

Sprint: Replication 2021-11-29, Replication 2021-12-13, Replication 2021-12-27
Participants:

 Description   

When we start mongod with `--profile --verbose` then rs.initiate() fails with "invariant failure" and mongod terminates after backtrace. If we remove one or both of those parameters then rs.initiate() succeeds.
See an example of "invariant failure" from the server log below:

{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"I",  "c":"REPL",     "id":21356,   "ctx":"conn1","msg":"replSetInitiate admin command received from client"}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn1","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":233}}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn1","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn1","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-10-26T19:24:40.009+03:00"},"s":"D1", "c":"-",        "id":23074,   "ctx":"ftdc","msg":"User assertion","attr":{"error":"NotYetInitialized: no replset config has been received","file":"src/mongo/db/repl/repl_set_get_status_cmd.cpp","line":56}}
{"t":{"$date":"2021-10-26T19:24:40.142+03:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn1","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"55A94A1ED135","b":"55A9463F4000","o":"3DF9135","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"55A94A1EFBC9","b":"55A9463F4000","o":"3DFBBC9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55A94A1E8206","b":"55A9463F4000","o":"3DF4206","s":"abruptQuit","s+":"66"},{"a":"7F652E4013C0","b":"7F652E3EC000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"7F652E24018B","b":"7F652E1FA000","o":"4618B","s":"gsignal","s+":"CB"},{"a":"7F652E21F859","b":"7F652E1FA000","o":"25859","s":"abort","s+":"12B"},{"a":"55A94778546A","b":"55A9463F4000","o":"139146A","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"55A94774A700","b":"55A9463F4000","o":"1356700","s":"_ZNK5mongo11FeatureFlag9isEnabledERKNS_18ServerGlobalParams20FeatureCompatibilityE.cold.73","s+":"18"},{"a":"55A9488780E0","b":"55A9463F4000","o":"24840E0","s":"_ZN5mongo4repl13ReplSetConfig29setDefaultDelayFieldForMemberENS0_12MemberConfigE.part.811","s+":"20"},{"a":"55A94887A02B","b":"55A9463F4000","o":"248602B","s":"_ZN5mongo4repl13ReplSetConfig11_initializeEbN5boost8optionalIxEENS_3OIDE","s+":"24B"},{"a":"55A94887AA0B","b":"55A9463F4000","o":"2486A0B","s":"_ZN5mongo4repl13ReplSetConfigC1ERKNS_7BSONObjEbN5boost8optionalIxEENS_3OIDE","s+":"24B"},{"a":"55A94887AD33","b":"55A9463F4000","o":"2486D33","s":"_ZN5mongo4repl13ReplSetConfig16parseForInitiateERKNS_7BSONObjENS_3OIDE","s+":"93"},{"a":"55A947CB22BB","b":"55A9463F4000","o":"18BE2BB","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl22processReplSetInitiateEPNS_16OperationContextERKNS_7BSONObjEPNS_14BSONObjBuilderE","s+":"22B"},{"a":"55A947C04174","b":"55A9463F4000","o":"1810174","s":"_ZN5mongo4repl18CmdReplSetInitiate3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE","s+":"8D4"},{"a":"55A947BFDD28","b":"55A9463F4000","o":"1809D28","s":"_ZN5mongo12BasicCommand19runWithReplyBuilderEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjEPNS_3rpc21ReplyBuilderInterfaceE","s+":"58"},{"a":"55A94903D0FE","b":"55A9463F4000","o":"2C490FE","s":"_ZN5mongo37BasicCommandWithReplyBuilderInterface10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"BE"},{"a":"55A949035C5F","b":"55A9463F4000","o":"2C41C5F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},{"a":"55A94903AEFE","b":"55A9463F4000","o":"2C46EFE","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","s+":"1BE"},{"a":"55A94792F257","b":"55A9463F4000","o":"153B257","s":"_ZN5mongo12_GLOBAL__N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","s+":"97"},{"a":"55A947941676","b":"55A9463F4000","o":"154D676","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl11_runCommandEv","s+":"146"},{"a":"55A9479426DB","b":"55A9463F4000","o":"154E6DB","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl8_runImplEv","s+":"8B"},{"a":"55A9479338AC","b":"55A9463F4000","o":"153F8AC","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl3runEv","s+":"13C"},{"a":"55A947935ABE","b":"55A9463F4000","o":"1541ABE","s":"_ZN5mongo12_GLOBAL__N_119ExecCommandDatabase12_commandExecEv","s+":"33E"},{"a":"55A94793E320","b":"55A9463F4000","o":"154A320","s":"_ZZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","s+":"450"},{"a":"55A947940510","b":"55A9463F4000","o":"154C510","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s+":"80"},{"a":"55A9478FB0E7","b":"55A9463F4000","o":"15070E7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","s+":"147"},{"a":"55A9479448D2","b":"55A9463F4000","o":"15508D2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1_","s+":"152"},{"a":"55A947937F5C","b":"55A9463F4000","o":"1543F5C","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s+":"AC"},{"a":"55A9478FB0E7","b":"55A9463F4000","o":"15070E7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","s+":"147"},{"a":"55A94793ED0A","b":"55A9463F4000","o":"154AD0A","s":"_ZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","s+":"22A"},{"a":"55A94793F73E","b":"55A9463F4000","o":"154B73E","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","s+":"1BE"},{"a":"55A947940456","b":"55A9463F4000","o":"154C456","s":"_ZN5mongo12_GLOBAL__N_115CommandOpRunner3runEv","s+":"36"},{"a":"55A94793A947","b":"55A9463F4000","o":"1546947","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE","s+":"307"},{"a":"55A94792A7DF","b":"55A9463F4000","o":"15367DF","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"4F"},{"a":"55A947982837","b":"55A9463F4000","o":"158E837","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","s+":"127"},{"a":"55A947982B67","b":"55A9463F4000","o":"158EB67","s":"_ZZNO5mongo14future_details10FutureImplINS0_8FakeVoidEE4thenIZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlvE0_EEDaOT_ENKUlOS2_E_clESE_.isra.718","s+":"27"},{"a":"55A947985498","b":"55A9463F4000","o":"1591498","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"1A8"},{"a":"55A94798599F","b":"55A9463F4000","o":"159199F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"55A9499CBCF1","b":"55A9463F4000","o":"35D7CF1","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"41"},{"a":"55A948014083","b":"55A9463F4000","o":"1C20083","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"55A9499CBEAB","b":"55A9463F4000","o":"35D7EAB","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"55A9499CFD5C","b":"55A9463F4000","o":"35DBD5C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"55A9499CFDCC","b":"55A9463F4000","o":"35DBDCC","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7F652E3F5609","b":"7F652E3EC000","o":"9609","s":"start_thread","s+":"D9"},{"a":"7F652E31C293","b":"7F652E1FA000","o":"122293","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.3","gitVersion":"657fea5a61a74d7a79df7aff8e4bcf0bc742b748","compiledModules":[],"uname":{"sysname":"Linux","release":"5.11.0-38-generic","version":"#42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021","machine":"x86_64"},"somap":[{"b":"55A9463F4000","elfType":3,"buildId":"599A16F3858E6AA2EB083B0CA9E2A1A4FD2CAB13"},{"b":"7F652E3EC000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"E54761F7B554D0FCC1562959665D93DFFBEBDAF0"},{"b":"7F652E1FA000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"099B9225BCB0D019D9D60884BE583EB31BB5F44E"}]}}}}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"I",  "c":"REPL",     "id":21356,   "ctx":"conn1","msg":"replSetInitiate admin command received from client"}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn1","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":233}}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn1","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-10-26T19:24:39.986+03:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn1","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-10-26T19:24:40.009+03:00"},"s":"D1", "c":"-",        "id":23074,   "ctx":"ftdc","msg":"User assertion","attr":{"error":"NotYetInitialized: no replset config has been received","file":"src/mongo/db/repl/repl_set_get_status_cmd.cpp","line":56}}
{"t":{"$date":"2021-10-26T19:24:40.142+03:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"conn1","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"55A94A1ED135","b":"55A9463F4000","o":"3DF9135","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357","s+":"215"},{"a":"55A94A1EFBC9","b":"55A9463F4000","o":"3DFBBC9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55A94A1E8206","b":"55A9463F4000","o":"3DF4206","s":"abruptQuit","s+":"66"},{"a":"7F652E4013C0","b":"7F652E3EC000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"7F652E24018B","b":"7F652E1FA000","o":"4618B","s":"gsignal","s+":"CB"},{"a":"7F652E21F859","b":"7F652E1FA000","o":"25859","s":"abort","s+":"12B"},{"a":"55A94778546A","b":"55A9463F4000","o":"139146A","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"F7"},{"a":"55A94774A700","b":"55A9463F4000","o":"1356700","s":"_ZNK5mongo11FeatureFlag9isEnabledERKNS_18ServerGlobalParams20FeatureCompatibilityE.cold.73","s+":"18"},{"a":"55A9488780E0","b":"55A9463F4000","o":"24840E0","s":"_ZN5mongo4repl13ReplSetConfig29setDefaultDelayFieldForMemberENS0_12MemberConfigE.part.811","s+":"20"},{"a":"55A94887A02B","b":"55A9463F4000","o":"248602B","s":"_ZN5mongo4repl13ReplSetConfig11_initializeEbN5boost8optionalIxEENS_3OIDE","s+":"24B"},{"a":"55A94887AA0B","b":"55A9463F4000","o":"2486A0B","s":"_ZN5mongo4repl13ReplSetConfigC1ERKNS_7BSONObjEbN5boost8optionalIxEENS_3OIDE","s+":"24B"},{"a":"55A94887AD33","b":"55A9463F4000","o":"2486D33","s":"_ZN5mongo4repl13ReplSetConfig16parseForInitiateERKNS_7BSONObjENS_3OIDE","s+":"93"},{"a":"55A947CB22BB","b":"55A9463F4000","o":"18BE2BB","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl22processReplSetInitiateEPNS_16OperationContextERKNS_7BSONObjEPNS_14BSONObjBuilderE","s+":"22B"},{"a":"55A947C04174","b":"55A9463F4000","o":"1810174","s":"_ZN5mongo4repl18CmdReplSetInitiate3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE","s+":"8D4"},{"a":"55A947BFDD28","b":"55A9463F4000","o":"1809D28","s":"_ZN5mongo12BasicCommand19runWithReplyBuilderEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjEPNS_3rpc21ReplyBuilderInterfaceE","s+":"58"},{"a":"55A94903D0FE","b":"55A9463F4000","o":"2C490FE","s":"_ZN5mongo37BasicCommandWithReplyBuilderInterface10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"BE"},{"a":"55A949035C5F","b":"55A9463F4000","o":"2C41C5F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},{"a":"55A94903AEFE","b":"55A9463F4000","o":"2C46EFE","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","s+":"1BE"},{"a":"55A94792F257","b":"55A9463F4000","o":"153B257","s":"_ZN5mongo12_GLOBAL__N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","s+":"97"},{"a":"55A947941676","b":"55A9463F4000","o":"154D676","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl11_runCommandEv","s+":"146"},{"a":"55A9479426DB","b":"55A9463F4000","o":"154E6DB","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl8_runImplEv","s+":"8B"},{"a":"55A9479338AC","b":"55A9463F4000","o":"153F8AC","s":"_ZN5mongo12_GLOBAL__N_114RunCommandImpl3runEv","s+":"13C"},{"a":"55A947935ABE","b":"55A9463F4000","o":"1541ABE","s":"_ZN5mongo12_GLOBAL__N_119ExecCommandDatabase12_commandExecEv","s+":"33E"},{"a":"55A94793E320","b":"55A9463F4000","o":"154A320","s":"_ZZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","s+":"450"},{"a":"55A947940510","b":"55A9463F4000","o":"154C510","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s+":"80"},{"a":"55A9478FB0E7","b":"55A9463F4000","o":"15070E7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","s+":"147"},{"a":"55A9479448D2","b":"55A9463F4000","o":"15508D2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1_","s+":"152"},{"a":"55A947937F5C","b":"55A9463F4000","o":"1543F5C","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s+":"AC"},{"a":"55A9478FB0E7","b":"55A9463F4000","o":"15070E7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","s+":"147"},{"a":"55A94793ED0A","b":"55A9463F4000","o":"154AD0A","s":"_ZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","s+":"22A"},{"a":"55A94793F73E","b":"55A9463F4000","o":"154B73E","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","s+":"1BE"},{"a":"55A947940456","b":"55A9463F4000","o":"154C456","s":"_ZN5mongo12_GLOBAL__N_115CommandOpRunner3runEv","s+":"36"},{"a":"55A94793A947","b":"55A9463F4000","o":"1546947","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE","s+":"307"},{"a":"55A94792A7DF","b":"55A9463F4000","o":"15367DF","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"4F"},{"a":"55A947982837","b":"55A9463F4000","o":"158E837","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","s+":"127"},{"a":"55A947982B67","b":"55A9463F4000","o":"158EB67","s":"_ZZNO5mongo14future_details10FutureImplINS0_8FakeVoidEE4thenIZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEEUlvE0_EEDaOT_ENKUlOS2_E_clESE_.isra.718","s+":"27"},{"a":"55A947985498","b":"55A9463F4000","o":"1591498","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","s+":"1A8"},{"a":"55A94798599F","b":"55A9463F4000","o":"159199F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"7F"},{"a":"55A9499CBCF1","b":"55A9463F4000","o":"35D7CF1","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s+":"41"},{"a":"55A948014083","b":"55A9463F4000","o":"1C20083","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","s+":"33"},{"a":"55A9499CBEAB","b":"55A9463F4000","o":"35D7EAB","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","s+":"BB"},{"a":"55A9499CFD5C","b":"55A9463F4000","o":"35DBD5C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","s+":"5C"},{"a":"55A9499CFDCC","b":"55A9463F4000","o":"35DBDCC","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"1C"},{"a":"7F652E3F5609","b":"7F652E3EC000","o":"9609","s":"start_thread","s+":"D9"},{"a":"7F652E31C293","b":"7F652E1FA000","o":"122293","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"5.0.3","gitVersion":"657fea5a61a74d7a79df7aff8e4bcf0bc742b748","compiledModules":[],"uname":{"sysname":"Linux","release":"5.11.0-38-generic","version":"#42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021","machine":"x86_64"},"somap":[{"b":"55A9463F4000","elfType":3,"buildId":"599A16F3858E6AA2EB083B0CA9E2A1A4FD2CAB13"},{"b":"7F652E3EC000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"E54761F7B554D0FCC1562959665D93DFFBEBDAF0"},{"b":"7F652E1FA000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"099B9225BCB0D019D9D60884BE583EB31BB5F44E"}]}}}}



 Comments   
Comment by Githook User [ 05/Jan/22 ]

Author:

{'name': 'Huayu Ouyang', 'email': 'huayu.ouyang@mongodb.com', 'username': 'huayu-ouyang'}

Message: SERVER-61005 rs.initiate() fails with "Invariant failure" under specific startup options

(cherry picked from commit 684d193982c2b53821f74c86cce76d88e90c5b15)
Branch: v5.0
https://github.com/mongodb/mongo/commit/71f035d8baf7d7827ad52981a81ac78951210db4

Comment by Huayu Ouyang [ 15/Dec/21 ]

igorsol It's been requested but not approved yet

Comment by Igor Solodovnikov [ 15/Dec/21 ]

Hello @Huayu Ouyang,
Thank you for the fix.
I have a question on this: is backport to v5.0 planned?

Comment by Githook User [ 13/Dec/21 ]

Author:

{'name': 'Huayu Ouyang', 'email': 'huayu.ouyang@mongodb.com', 'username': 'huayu-ouyang'}

Message: SERVER-61005 rs.initiate() fails with "Invariant failure" under specific startup options
Branch: master
https://github.com/mongodb/mongo/commit/684d193982c2b53821f74c86cce76d88e90c5b15

Comment by Huayu Ouyang [ 18/Nov/21 ]

The reason the FCV isn't initialized is because when we try to initialize it here , we check if we are in clean startup by checking if there are any other databases other than "local".

For some reason, when we start with this specific combination of startup options, there ends up being two databases, "admin" and "local".

Comment by Huayu Ouyang [ 18/Nov/21 ]

It looks like the invariant is happening in this feature compatibility version check, which was removed in 5.1

Comment by Huayu Ouyang [ 18/Nov/21 ]

I tried reproducing this with a jstest, and it seems like it only occurs when we add the auth and clusterAuthMode=keyFile/ keyFile flags as well. 

Edit: It actually only occurs when we specifically add the keyFile flag

Comment by Edwin Zhou [ 26/Oct/21 ]

symbolized backtrace:

Ignoring frame {'a': '700001FED898'} as it's missing the `b` field; See SERVER-58863 for discussions
??:0:0: mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*)
??:0:0: mongo::printStackTrace()
??:0:0: abruptQuit
??:0:0: ??
??:0:0: ??
??:0:0: mongo::invariantFailed(char const*, char const*, unsigned int)
??:0:0: mongo::FeatureFlag::isEnabled(mongo::ServerGlobalParams::FeatureCompatibility const&) const
??:0:0: mongo::repl::ReplSetConfig::_initialize(bool, boost::optional<long long>, mongo::OID)
??:0:0: mongo::repl::ReplSetConfig::ReplSetConfig(mongo::BSONObj const&, bool, boost::optional<long long>, mongo::OID)
??:0:0: mongo::repl::ReplSetConfig::parseForInitiate(mongo::BSONObj const&, mongo::OID)
??:0:0: mongo::repl::ReplicationCoordinatorImpl::processReplSetInitiate(mongo::OperationContext*, mongo::BSONObj const&, mongo::BSONObjBuilder*)
??:0:0: mongo::repl::CmdReplSetInitiate::run(mongo::OperationContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&)
??:0:0: mongo::BasicCommand::runWithReplyBuilder(mongo::OperationContext*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)
??:0:0: mongo::BasicCommandWithReplyBuilderInterface::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
??:0:0: mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)
??:0:0: mongo::CommandHelpers::runCommandInvocation(std::__1::shared_ptr<mongo::RequestExecutionContext>, std::__1::shared_ptr<mongo::CommandInvocation>, mongo::transport::ServiceExecutor::ThreadingModel)
??:0:0: mongo::(anonymous namespace)::runCommandInvocation(std::__1::shared_ptr<mongo::RequestExecutionContext>, std::__1::shared_ptr<mongo::CommandInvocation>)
??:0:0: mongo::(anonymous namespace)::RunCommandImpl::_runCommand()
??:0:0: mongo::(anonymous namespace)::RunCommandImpl::_runImpl()
??:0:0: mongo::(anonymous namespace)::RunCommandImpl::run()
??:0:0: mongo::(anonymous namespace)::ExecCommandDatabase::_commandExec()
??:0:0: mongo::(anonymous namespace)::ExecCommandDatabase::run()
??:0:0: auto mongo::future_details::throwingCall<mongo::(anonymous namespace)::executeCommand(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_25&, mongo::future_details::FakeVoid>(mongo::(anonymous namespace)::executeCommand(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_25&, mongo::future_details::FakeVoid&&)
??:0:0: _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandENSt3__110shared_ptrINSC_13HandleRequest16ExecutionContextEEEE4$_25EEDaOT_ENKUlvE_clEvEUlPNS1_15SharedStateImplIS8_EESP_E_EENS7_ISK_EEOT0_EUlS3_E_EESJ_SL_EN12SpecificImpl4callEOS3_
??:0:0: mongo::future_details::SharedStateBase::transitionToFinished()
??:0:0: _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandENSt3__110shared_ptrINSC_13HandleRequest16ExecutionContextEEEE4$_24EEDaOT_ENKUlvE_clEvEUlPNS1_15SharedStateImplIS8_EESP_E_EENS7_ISK_EEOT0_EUlS3_E_EESJ_SL_EN12SpecificImpl4callEOS3_
??:0:0: mongo::future_details::SharedStateBase::transitionToFinished()
??:0:0: auto mongo::future_details::throwingCall<mongo::(anonymous namespace)::receivedCommands(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_27&, mongo::future_details::FakeVoid>(mongo::(anonymous namespace)::receivedCommands(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_27&, mongo::future_details::FakeVoid&&)
??:0:0: auto mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::receivedCommands(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_27>(mongo::(anonymous namespace)::receivedCommands(std::__1::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::$_27&&) &&::'lambda'(mongo::future_details::FakeVoid&&)::operator()('lambda'(mongo::future_details::FakeVoid&&)) const
??:0:0: mongo::(anonymous namespace)::CommandOpRunner::run()
??:0:0: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, std::__1::unique_ptr<mongo::ServiceEntryPointCommon::Hooks const, std::__1::default_delete<mongo::ServiceEntryPointCommon::Hooks const> >)
??:0:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
??:0:0: mongo::transport::ServiceStateMachine::Impl::processMessage()
??:0:0: auto mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)::$_10>(mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)::$_10&&) &&::'lambda'(mongo::future_details::FakeVoid&&)::operator()('lambda'(mongo::future_details::FakeVoid&&)) const
??:0:0: mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)
??:0:0: auto mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::$_7>(mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::$_7&&)::SpecificImpl::call(mongo::Status&&)
??:0:0: auto mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::__1::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::$_5>(mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::__1::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::$_5&&)::SpecificImpl::call(mongo::Status&&)
??:0:0: mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()::operator()()
??:0:0: auto mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_4>(mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::$_4&&)::SpecificImpl::call()
??:0:0: mongo::(anonymous namespace)::runFunc(void*)
??:0:0: ??
??:0:0: ??

Comment by Edwin Zhou [ 26/Oct/21 ]

Hi igorsol,

Thanks for reporting this issue and for your detailed and concise description and reproduction. I've been able to reproduce this issue on my end and will pass this on to the replication team for further investigation.

Best,
Edwin

Generated at Thu Feb 08 05:51:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.