[SERVER-72775] Issue upgrading from 5.0.14 to 6.x Created: 12/Jan/23 Updated: 13/Jan/23 Resolved: 13/Jan/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 6.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Alessandro Lombardi | Assignee: | Edwin Zhou |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | upgrade from 5.0.14 to 6.0.x |
||||||||
| Participants: | |||||||||
| Description |
we have this issue upgrading 2 different mongo instances, 1 is a standalone and another replicaset with 3 nodes. FCV set to 5.0 We can't understand why it blocks. |
| Comments |
| Comment by Alessandro Lombardi [ 13/Jan/23 ] |
|
Thank you Edwin for your investigation. We will wait for the fix to try the upgrade again. Best Regards, Alessandro |
| Comment by Edwin Zhou [ 13/Jan/23 ] |
|
Hi alessandro.lombardi@coreview.com, After investigating this invariant, the issue you're hitting is Best, |
| Comment by Edwin Zhou [ 12/Jan/23 ] |
|
Thank you for uploading the log, I can confirm that I'm able to open the log file. I will pass this ticket on to the appropriate team to further investigate this invariant failure on startup. |
| Comment by Alessandro Lombardi [ 12/Jan/23 ] |
|
yes exactly, this is what I mean. I have uploaded now the log to the link you provide me, let me know if you can open the file. |
| Comment by Edwin Zhou [ 12/Jan/23 ] |
|
Hi alessandro.lombardi@coreview.com, Thank you for the swift follow up and the additional information! Since this invariant appears to be hit on startup, could you attach a single occurrence of the log from startup to when mongod hits the invariant failure? |
| Comment by Alessandro Lombardi [ 12/Jan/23 ] |
|
ok I try to upload the file to you but is a loop of the messages posted above. The 2 instances where we have the problem are:
the log shows the same messages. FCV is correcly set and the answer is ok { "featureCompatibilityVersion" : \{ "version" : "5.0" }, "ok" : 1 } we have another docker standalone where the upgrade was successful, same config of the other one. I believe that the problem could be on data but I don't know what to check. Rollback to 5.0.14 works, issue was also present upgrading to 6.x from 5.0.8. This mongo data are coming from old 4.x versions where we have not experienced any issue upgrading to 5.x |
| Comment by Edwin Zhou [ 12/Jan/23 ] |
|
The invariant seems to be coming from a feature flag check during startup recovery, where FCV does not seem to have been set yet. |
| Comment by Edwin Zhou [ 12/Jan/23 ] |
|
Hi alessandro.lombardi@coreview.com, Thank you for submitting this incident. Would you please archive (tar or zip) the mongod.log files for this incident and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Kind regards, |
| Comment by Alessandro Lombardi [ 12/Jan/23 ] |
|
{"t":\{"$date":"2023-01-11T08:32:59.046+00:00"}
,"s":"W", "c":"CONTROL", "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"ASSERT", "id":23079, "ctx":"initandlisten","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":156}} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"ASSERT", "id":23080, "ctx":"initandlisten","msg":"\n\n***aborting after invariant() failure\n\n"} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"CONTROL", "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t":\{"$date":"2023-01-11T08:33:25.236+00:00"},"s":"I", "c":"CONTROL", "id":31380, "ctx":"initandlisten","msg":"BACKTRACE","attr":{"bt":{"backtrace":[ {"a":"55ACD2015094","b":"55ACCD247000","o":"4DCE094","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"},{"a":"55ACD20175D9","b":"55ACCD247000","o":"4DD05D9","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55ACD20112B6","b":"55ACCD247000","o":"4DCA2B6","s":"abruptQuit","s+":"66"},{"a":"7FE7D737F420","b":"7FE7D736B000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7FE7D71BC00B","b":"7FE7D7179000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7FE7D719B859","b":"7FE7D7179000","o":"22859","s":"abort","s+":"12B"},{"a":"55ACCF13FDDA","b":"55ACCD247000","o":"1EF8DDA","s":"_ZN5mongo15invariantFailedEPKcS1_j","C":"mongo::invariantFailed(char const*, char const*, unsigned int)","s+":"F7"},{"a":"55ACCF1096D8","b":"55ACCD247000","o":"1EC26D8","s":"_ZNK5mongo11FeatureFlag9isEnabledERKNS_18ServerGlobalParams20FeatureCompatibilityE.cold.76","C":"mongo::FeatureFlag::isEnabled(mongo::ServerGlobalParams::FeatureCompatibility const&) const [clone .cold.76]","s+":"18"},{"a":"55ACD02471E7","b":"55ACCD247000","o":"30001E7","s":"_ZNK5mongo16IndexCatalogImpl9_isSpecOkEPNS_16OperationContextERKNS_13CollectionPtrERKNS_7BSONObjE","C":"mongo::IndexCatalogImpl::_isSpecOk(mongo::OperationContext*, mongo::CollectionPtr const&, mongo::BSONObj const&) const","s+":"8C7"},{"a":"55ACD024B174","b":"55ACCD247000","o":"3004174","s":"_ZN5mongo16IndexCatalogImpl16createIndexEntryEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_15IndexDescriptorESt14default_deleteIS6_EENS_21CreateIndexEntryFlagsE","C":"mongo::IndexCatalogImpl::createIndexEntry(mongo::OperationContext*, mongo::Collection*, std::unique_ptr<mongo::IndexDescriptor, std::default_delete<mongo::IndexDescriptor> >, mongo::CreateIndexEntryFlags)","s+":"84"},{"a":"55ACD024ADA8","b":"55ACCD247000","o":"3003DA8","s":"_ZN5mongo16IndexCatalogImpl4initEPNS_16OperationContextEPNS_10CollectionE","C":"mongo::IndexCatalogImpl::init(mongo::OperationContext*, mongo::Collection*)","s+":"848"},{"a":"55ACD021E2C5","b":"55ACCD247000","o":"2FD72C5","s":"_ZN5mongo14CollectionImpl4initEPNS_16OperationContextE","C":"mongo::CollectionImpl::init(mongo::OperationContext*)","s+":"3C5"},{"a":"55ACD02377A1","b":"55ACCD247000","o":"2FF07A1","s":"_ZN5mongo12DatabaseImpl4initEPNS_16OperationContextE","C":"mongo::DatabaseImpl::init(mongo::OperationContext*)","s+":"121"},{"a":"55ACD0231DAF","b":"55ACCD247000","o":"2FEADAF","s":"_ZN5mongo18DatabaseHolderImpl6openDbEPNS_16OperationContextERKNS_18TenantDatabaseNameEPb","C":"mongo::DatabaseHolderImpl::openDb(mongo::OperationContext*, mongo::TenantDatabaseName const&, bool*)","s+":"30F"},{"a":"55ACD07549C6","b":"55ACCD247000","o":"350D9C6","s":"_ZN5mongo9AutoGetDb14ensureDbExistsEPNS_16OperationContextE","C":"mongo::AutoGetDb::ensureDbExists(mongo::OperationContext*)","s+":"B6"},{"a":"55ACD042068E","b":"55ACCD247000","o":"31D968E","s":"_ZN5mongo27FeatureCompatibilityVersion39findFeatureCompatibilityVersionDocumentEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::findFeatureCompatibilityVersionDocument(mongo::OperationContext*)","s+":"20E"},{"a":"55ACD0420F4C","b":"55ACCD247000","o":"31D9F4C","s":"_ZN5mongo27FeatureCompatibilityVersion20initializeForStartupEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::initializeForStartup(mongo::OperationContext*)","s+":"4C"},{"a":"55ACCF39A135","b":"55ACCD247000","o":"2153135","s":"_ZN5mongo12_GLOBAL_N_115startupRecoveryEPNS_16OperationContextEPNS_13StorageEngineENS3_17LastShutdownStateENS_16startup_recovery19StartupRecoveryModeE","C":"mongo::(anonymous namespace)::startupRecovery(mongo::OperationContext*, mongo::StorageEngine*, mongo::StorageEngine::LastShutdownState, mongo::startup_recovery::StartupRecoveryMode)","s+":"55"},{"a":"55ACCF39A8E9","b":"55ACCD247000","o":"21538E9","s":"ZN5mongo16startup_recovery25repairAndRecoverDatabasesEPNS_16OperationContextENS_13StorageEngine17LastShutdownStateE","C":"mongo::startup_recovery::repairAndRecoverDatabases(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState)","s+":"269"},{"a":"55ACCF35DC91","b":"55ACCD247000","o":"2116C91","s":"_ZN5mongo12_GLOBAL_N_114_initAndListenEPNS_14ServiceContextEi.isra.2009","C":"mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) [clone .isra.2009]","s+":"A81"},{"a":"55ACCF360EED","b":"55ACCD247000","o":"2119EED","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"BDD"},{"a":"55ACCF150FCE","b":"55ACCD247000","o":"1F09FCE","s":"main","s+":"E"},{"a":"7FE7D719D083","b":"7FE7D7179000","o":"24083","s":"_libc_start_main","s+":"F3"},{"a":"55ACCF35B74E","b":"55ACCD247000","o":"211474E","s":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"6.0.3","gitVersion":"f803681c3ae19817d31958965850193de067c516","compiledModules":[],"uname": {"sysname":"Linux","release":"4.4.0-186-generic","version":"#216-Ubuntu SMP Wed Jul 1 05:34:05 UTC 2020","machine":"x86_64"},"somap":[\{"b":"55ACCD247000","elfType":3,"buildId":"1D29859BFD681BF978651663140EBA623DD9D6E7"},\{"b":"7FE7D736B000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},\{"b":"7FE7D7179000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}},"tags":[]} |