[SERVER-72775] Issue upgrading from 5.0.14 to 6.x Created: 12/Jan/23  Updated: 13/Jan/23  Resolved: 13/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alessandro Lombardi Assignee: Edwin Zhou
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-71068 Partial indexes should not check FCV ... Closed
Operating System: ALL
Steps To Reproduce:

upgrade from 5.0.14 to 6.0.x

Participants:

 Description   

// code placeholder
{"t":{"$date":"2023-01-11T08:32:59.046+00:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F",  "c":"ASSERT",   "id":23079,   "ctx":"initandlisten","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":156}}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F",  "c":"ASSERT",   "id":23080,   "ctx":"initandlisten","msg":"\n\n***aborting after invariant() failure\n\n"}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} 

we have this issue upgrading 2 different mongo instances, 1 is a standalone and another replicaset with 3 nodes.

FCV set to 5.0

We can't understand why it blocks.



 Comments   
Comment by Alessandro Lombardi [ 13/Jan/23 ]

Thank you Edwin for your investigation.

We will wait for the fix to try the upgrade again.

Best Regards,

Alessandro

Comment by Edwin Zhou [ 13/Jan/23 ]

Hi alessandro.lombardi@coreview.com,

After investigating this invariant, the issue you're hitting is SERVER-71068, where a mongod on 6.0 with FCV set to 5.0 will fail to start up when the server is checking whether the index spec is valid. SERVER-71068 should fix this problem by not checking FCV during startup. I will close this as a duplicate of SERVER-71068, which you may watch for updates.

Best,
Edwin

Comment by Edwin Zhou [ 12/Jan/23 ]

Thank you for uploading the log, I can confirm that I'm able to open the log file.

I will pass this ticket on to the appropriate team to further investigate this invariant failure on startup.

Comment by Alessandro Lombardi [ 12/Jan/23 ]

yes exactly, this is what I mean.

I have uploaded now the log to the link you provide me, let me know if you can open the file.

Comment by Edwin Zhou [ 12/Jan/23 ]

Hi alessandro.lombardi@coreview.com,

Thank you for the swift follow up and the additional information!
I presume when you say, "it is a loop of the messages posted above" that you mean that on every attempt to start the mongod on 6.0, the aborts when hitting this invariant.

Since this invariant appears to be hit on startup, could you attach a single occurrence of the log from startup to when mongod hits the invariant failure?

Comment by Alessandro Lombardi [ 12/Jan/23 ]

ok I try to upload the file to you but is a loop of the messages posted above.

The 2 instances where we have the problem are:

  • one standalone in docker
  • one replicaset managed by cloud manager (where we have open a ticket to support currently in progress)

the log shows the same messages.

FCV is correcly set and the answer is ok 

{ "featureCompatibilityVersion" : \{ "version" : "5.0" }

, "ok" : 1 }

we have another docker standalone where the upgrade was successful, same config of the other one.

I believe that the problem could be on data but I don't know what to check. Rollback to 5.0.14 works, issue was also present upgrading to 6.x from 5.0.8.

This mongo data are coming from old 4.x versions where we have not experienced any issue upgrading to 5.x

Comment by Edwin Zhou [ 12/Jan/23 ]

The invariant seems to be coming from a feature flag check during startup recovery, where FCV does not seem to have been set yet.
https://github.com/mongodb/mongo/blob/r6.0.3/src/mongo/db/server_options.h#L147-L158

Comment by Edwin Zhou [ 12/Jan/23 ]

Hi alessandro.lombardi@coreview.com,

Thank you for submitting this incident.

Would you please archive (tar or zip) the mongod.log files for this incident and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Kind regards,
Edwin

Comment by Alessandro Lombardi [ 12/Jan/23 ]
{"t":\{"$date":"2023-01-11T08:32:59.046+00:00"}

,"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}

{"t":\{"$date":"2023-01-11T08:32:59.060+00:00"}

,"s":"F",  "c":"ASSERT",   "id":23079,   "ctx":"initandlisten","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":156}}

{"t":\{"$date":"2023-01-11T08:32:59.060+00:00"}

,"s":"F",  "c":"ASSERT",   "id":23080,   "ctx":"initandlisten","msg":"\n\n***aborting after invariant() failure\n\n"}

{"t":\{"$date":"2023-01-11T08:32:59.060+00:00"}

,"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

{"t":\{"$date":"2023-01-11T08:33:25.236+00:00"}

,"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"initandlisten","msg":"BACKTRACE","attr":{"bt":{"backtrace":[

{"a":"55ACD2015094","b":"55ACCD247000","o":"4DCE094","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}

,{"a":"55ACD20175D9","b":"55ACCD247000","o":"4DD05D9","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55ACD20112B6","b":"55ACCD247000","o":"4DCA2B6","s":"abruptQuit","s+":"66"},{"a":"7FE7D737F420","b":"7FE7D736B000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7FE7D71BC00B","b":"7FE7D7179000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7FE7D719B859","b":"7FE7D7179000","o":"22859","s":"abort","s+":"12B"},{"a":"55ACCF13FDDA","b":"55ACCD247000","o":"1EF8DDA","s":"_ZN5mongo15invariantFailedEPKcS1_j","C":"mongo::invariantFailed(char const*, char const*, unsigned int)","s+":"F7"},{"a":"55ACCF1096D8","b":"55ACCD247000","o":"1EC26D8","s":"_ZNK5mongo11FeatureFlag9isEnabledERKNS_18ServerGlobalParams20FeatureCompatibilityE.cold.76","C":"mongo::FeatureFlag::isEnabled(mongo::ServerGlobalParams::FeatureCompatibility const&) const [clone .cold.76]","s+":"18"},{"a":"55ACD02471E7","b":"55ACCD247000","o":"30001E7","s":"_ZNK5mongo16IndexCatalogImpl9_isSpecOkEPNS_16OperationContextERKNS_13CollectionPtrERKNS_7BSONObjE","C":"mongo::IndexCatalogImpl::_isSpecOk(mongo::OperationContext*, mongo::CollectionPtr const&, mongo::BSONObj const&) const","s+":"8C7"},{"a":"55ACD024B174","b":"55ACCD247000","o":"3004174","s":"_ZN5mongo16IndexCatalogImpl16createIndexEntryEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_15IndexDescriptorESt14default_deleteIS6_EENS_21CreateIndexEntryFlagsE","C":"mongo::IndexCatalogImpl::createIndexEntry(mongo::OperationContext*, mongo::Collection*, std::unique_ptr<mongo::IndexDescriptor, std::default_delete<mongo::IndexDescriptor> >, mongo::CreateIndexEntryFlags)","s+":"84"},{"a":"55ACD024ADA8","b":"55ACCD247000","o":"3003DA8","s":"_ZN5mongo16IndexCatalogImpl4initEPNS_16OperationContextEPNS_10CollectionE","C":"mongo::IndexCatalogImpl::init(mongo::OperationContext*, mongo::Collection*)","s+":"848"},{"a":"55ACD021E2C5","b":"55ACCD247000","o":"2FD72C5","s":"_ZN5mongo14CollectionImpl4initEPNS_16OperationContextE","C":"mongo::CollectionImpl::init(mongo::OperationContext*)","s+":"3C5"},{"a":"55ACD02377A1","b":"55ACCD247000","o":"2FF07A1","s":"_ZN5mongo12DatabaseImpl4initEPNS_16OperationContextE","C":"mongo::DatabaseImpl::init(mongo::OperationContext*)","s+":"121"},{"a":"55ACD0231DAF","b":"55ACCD247000","o":"2FEADAF","s":"_ZN5mongo18DatabaseHolderImpl6openDbEPNS_16OperationContextERKNS_18TenantDatabaseNameEPb","C":"mongo::DatabaseHolderImpl::openDb(mongo::OperationContext*, mongo::TenantDatabaseName const&, bool*)","s+":"30F"},{"a":"55ACD07549C6","b":"55ACCD247000","o":"350D9C6","s":"_ZN5mongo9AutoGetDb14ensureDbExistsEPNS_16OperationContextE","C":"mongo::AutoGetDb::ensureDbExists(mongo::OperationContext*)","s+":"B6"},{"a":"55ACD042068E","b":"55ACCD247000","o":"31D968E","s":"_ZN5mongo27FeatureCompatibilityVersion39findFeatureCompatibilityVersionDocumentEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::findFeatureCompatibilityVersionDocument(mongo::OperationContext*)","s+":"20E"},{"a":"55ACD0420F4C","b":"55ACCD247000","o":"31D9F4C","s":"_ZN5mongo27FeatureCompatibilityVersion20initializeForStartupEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::initializeForStartup(mongo::OperationContext*)","s+":"4C"},{"a":"55ACCF39A135","b":"55ACCD247000","o":"2153135","s":"_ZN5mongo12_GLOBAL_N_115startupRecoveryEPNS_16OperationContextEPNS_13StorageEngineENS3_17LastShutdownStateENS_16startup_recovery19StartupRecoveryModeE","C":"mongo::(anonymous namespace)::startupRecovery(mongo::OperationContext*, mongo::StorageEngine*, mongo::StorageEngine::LastShutdownState, mongo::startup_recovery::StartupRecoveryMode)","s+":"55"},{"a":"55ACCF39A8E9","b":"55ACCD247000","o":"21538E9","s":"ZN5mongo16startup_recovery25repairAndRecoverDatabasesEPNS_16OperationContextENS_13StorageEngine17LastShutdownStateE","C":"mongo::startup_recovery::repairAndRecoverDatabases(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState)","s+":"269"},{"a":"55ACCF35DC91","b":"55ACCD247000","o":"2116C91","s":"_ZN5mongo12_GLOBAL_N_114_initAndListenEPNS_14ServiceContextEi.isra.2009","C":"mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) [clone .isra.2009]","s+":"A81"},{"a":"55ACCF360EED","b":"55ACCD247000","o":"2119EED","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"BDD"},{"a":"55ACCF150FCE","b":"55ACCD247000","o":"1F09FCE","s":"main","s+":"E"},{"a":"7FE7D719D083","b":"7FE7D7179000","o":"24083","s":"_libc_start_main","s+":"F3"},{"a":"55ACCF35B74E","b":"55ACCD247000","o":"211474E","s":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"6.0.3","gitVersion":"f803681c3ae19817d31958965850193de067c516","compiledModules":[],"uname":

{"sysname":"Linux","release":"4.4.0-186-generic","version":"#216-Ubuntu SMP Wed Jul 1 05:34:05 UTC 2020","machine":"x86_64"}

,"somap":[\{"b":"55ACCD247000","elfType":3,"buildId":"1D29859BFD681BF978651663140EBA623DD9D6E7"},\{"b":"7FE7D736B000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},\{"b":"7FE7D7179000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}},"tags":[]}

Generated at Thu Feb 08 06:22:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.