Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54876

Upgrade mongodb 4.4.4 from 4.2.12 failed

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 4.4.3, 4.4.4
    • None
    • Vmware virtual machines Centos 7.9.
      Replica set 7 nodes ( six db and one arbiter).
    • Replication
    • ALL
    • v4.4
    • Repl 2021-04-19, Repl 2021-05-03, Repl 2021-05-17

    Description

      Hi,

      we've a replica set with 6 nodes mongodb e one arbiter.

      I  tried upgrade from 4.2.12 to 4.4.4 (rolling upgrade) and i did the following steps: 

      1. Upgrade from 4.2.12 to 4.4.2 one secondary node and the node it works perfectly in secondary mode.
      2. switch primary from node 4.2.12 to 4.4.4 and the node 4.4.4 crashed.

      This is the output log trace before the node crash :

      {"t":\{"$date":"2021-03-02T08:08:19.684+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"conn1407","msg":"BACKTRACE: \{bt}","attr":\{"bt":{"backtrace":[{"a":"559B86781BE1","b":"559B83A79000","o":"2D08BE1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},\{"a":"559B86783219","b":"559B83A79000","o":"2D0A219","s":"_ZN5mongo15printStackTraceEv","s+":"29"},\{"a":"559B86780A76","b":"559B83A79000","o":"2D07A76","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},\{"a":"7F336A2F1630","b":"7F336A2E2000","o":"F630","s":"_L_unlock_13","s+":"34"},\{"a":"7F3369F4A3D7","b":"7F3369F14000","o":"363D7","s":"gsignal","s+":"37"},\{"a":"7F3369F4BAC8","b":"7F3369F14000","o":"37AC8","s":"abort","s+":"148"},\{"a":"559B84993677","b":"559B83A79000","o":"F1A677","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"},\{"a":"559B846E5560","b":"559B83A79000","o":"C6C560","s":"_ZN5mongo4repl10MemberData35advanceLastDurableOpTimeAndWallTimeENS0_17OpTimeAndWallTimeENS_6Date_tE.cold.338","s+":"18"},\{"a":"559B84E097D3","b":"559B83A79000","o":"13907D3","s":"_ZN5mongo4repl19TopologyCoordinator13setLastOptimeERKNS0_18UpdatePositionArgs10UpdateInfoENS_6Date_tEPx","s+":"2E3"},\{"a":"559B84D56C65","b":"559B83A79000","o":"12DDC65","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl14_setLastOptimeENS_8WithLockERKNS0_18UpdatePositionArgs10UpdateInfoEPx","s+":"55"},\{"a":"559B84D56DCF","b":"559B83A79000","o":"12DDDCF","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl28processReplSetUpdatePositionERKNS0_18UpdatePositionArgsEPx","s+":"AF"},\{"a":"559B84D11317","b":"559B83A79000","o":"1298317","s":"_ZN5mongo4repl24CmdReplSetUpdatePosition3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE","s+":"167"},\{"a":"559B856EC64A","b":"559B83A79000","o":"1C7364A","s":"_ZN5mongo37BasicCommandWithReplyBuilderInterface10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"11A"},\{"a":"559B856E6B8F","b":"559B83A79000","o":"1C6DB8F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},\{"a":"559B84E8E665","b":"559B83A79000","o":"1415665","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"135"},\{"a":"559B84E914C9","b":"559B83A79000","o":"14184C9","s":"_ZN5mongo12_GLOBAL__N_119execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceERKNS_23ServiceEntryPointCommon5HooksE","s+":"11B9"},\{"a":"559B84E927B0","b":"559B83A79000","o":"14197B0","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksE","s+":"650"},\{"a":"559B84E9343D","b":"559B83A79000","o":"141A43D","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"50D"},\{"a":"559B84E80EEC","b":"559B83A79000","o":"1407EEC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"},\{"a":"559B84E8B975","b":"559B83A79000","o":"1412975","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"F5"},\{"a":"559B84E89685","b":"559B83A79000","o":"1410685","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"},\{"a":"559B84E8A616","b":"559B83A79000","o":"1411616","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B86148162","b":"559B83A79000","o":"26CF162","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"182"},\{"a":"559B84E88A9B","b":"559B83A79000","o":"140FA9B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"},\{"a":"559B84E89DFD","b":"559B83A79000","o":"1410DFD","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"6AD"},\{"a":"559B84E89EE0","b":"559B83A79000","o":"1410EE0","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"},\{"a":"559B84E8A185","b":"559B83A79000","o":"1411185","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"},\{"a":"559B84E896FA","b":"559B83A79000","o":"14106FA","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"},\{"a":"559B84E8A616","b":"559B83A79000","o":"1411616","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B861487C8","b":"559B83A79000","o":"26CF7C8","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"B8"},\{"a":"559B864991E6","b":"559B83A79000","o":"2A201E6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B86499254","b":"559B83A79000","o":"2A20254","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"},\{"a":"7F336A2E9EA5","b":"7F336A2E2000","o":"7EA5","s":"start_thread","s+":"C5"},\{"a":"7F336A0129FD","b":"7F3369F14000","o":"FE9FD","s":"clone","s+":"6D"}],"processInfo":\{"mongodbVersion":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","compiledModules":[],"uname":{"sysname":"Linux","release":"3.10.0-1160.15.2.el7.x86_64","version":"#1 SMP Wed Feb 3 15:06:38 UTC 2021","machine":"x86_64"},"somap":[\{"b":"559B83A79000","elfType":3,"buildId":"CC93F86C6D00CE717DB23676747D258B4027E066"},\{"b":"7F336A2E2000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B"},\{"b":"7F3369F14000","path":"/lib64/libc.so.6","elfType":3,"buildId":"7CD4A08C18C60E1E2EA1AEBB88C9379BD7289D38"}]}}}} 
      

      I tried also with the version 4.4.3 but have the same errors.

       

      Attachments

        1. mongod.log.rpmsave.gz
          145 kB
        2. mongod.log.rpmsave.zip
          145 kB

        Activity

          People

            backlog-server-repl Backlog - Replication Team
            nicola.battista89@gmail.com Nicola Battista
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: