Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54876

Upgrade mongodb 4.4.4 from 4.2.12 failed

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.3, 4.4.4
    • Component/s: None
    • Environment:
      Vmware virtual machines Centos 7.9.
      Replica set 7 nodes ( six db and one arbiter).
    • Replication
    • ALL
    • v4.4
    • Repl 2021-04-19, Repl 2021-05-03, Repl 2021-05-17
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi,

      we've a replica set with 6 nodes mongodb e one arbiter.

      I  tried upgrade from 4.2.12 to 4.4.4 (rolling upgrade) and i did the following steps: 

      1. Upgrade from 4.2.12 to 4.4.2 one secondary node and the node it works perfectly in secondary mode.
      2. switch primary from node 4.2.12 to 4.4.4 and the node 4.4.4 crashed.

      This is the output log trace before the node crash :

      {"t":\{"$date":"2021-03-02T08:08:19.684+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"conn1407","msg":"BACKTRACE: \{bt}","attr":\{"bt":{"backtrace":[{"a":"559B86781BE1","b":"559B83A79000","o":"2D08BE1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},\{"a":"559B86783219","b":"559B83A79000","o":"2D0A219","s":"_ZN5mongo15printStackTraceEv","s+":"29"},\{"a":"559B86780A76","b":"559B83A79000","o":"2D07A76","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},\{"a":"7F336A2F1630","b":"7F336A2E2000","o":"F630","s":"_L_unlock_13","s+":"34"},\{"a":"7F3369F4A3D7","b":"7F3369F14000","o":"363D7","s":"gsignal","s+":"37"},\{"a":"7F3369F4BAC8","b":"7F3369F14000","o":"37AC8","s":"abort","s+":"148"},\{"a":"559B84993677","b":"559B83A79000","o":"F1A677","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"},\{"a":"559B846E5560","b":"559B83A79000","o":"C6C560","s":"_ZN5mongo4repl10MemberData35advanceLastDurableOpTimeAndWallTimeENS0_17OpTimeAndWallTimeENS_6Date_tE.cold.338","s+":"18"},\{"a":"559B84E097D3","b":"559B83A79000","o":"13907D3","s":"_ZN5mongo4repl19TopologyCoordinator13setLastOptimeERKNS0_18UpdatePositionArgs10UpdateInfoENS_6Date_tEPx","s+":"2E3"},\{"a":"559B84D56C65","b":"559B83A79000","o":"12DDC65","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl14_setLastOptimeENS_8WithLockERKNS0_18UpdatePositionArgs10UpdateInfoEPx","s+":"55"},\{"a":"559B84D56DCF","b":"559B83A79000","o":"12DDDCF","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl28processReplSetUpdatePositionERKNS0_18UpdatePositionArgsEPx","s+":"AF"},\{"a":"559B84D11317","b":"559B83A79000","o":"1298317","s":"_ZN5mongo4repl24CmdReplSetUpdatePosition3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE","s+":"167"},\{"a":"559B856EC64A","b":"559B83A79000","o":"1C7364A","s":"_ZN5mongo37BasicCommandWithReplyBuilderInterface10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"11A"},\{"a":"559B856E6B8F","b":"559B83A79000","o":"1C6DB8F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},\{"a":"559B84E8E665","b":"559B83A79000","o":"1415665","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"135"},\{"a":"559B84E914C9","b":"559B83A79000","o":"14184C9","s":"_ZN5mongo12_GLOBAL__N_119execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceERKNS_23ServiceEntryPointCommon5HooksE","s+":"11B9"},\{"a":"559B84E927B0","b":"559B83A79000","o":"14197B0","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksE","s+":"650"},\{"a":"559B84E9343D","b":"559B83A79000","o":"141A43D","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"50D"},\{"a":"559B84E80EEC","b":"559B83A79000","o":"1407EEC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"},\{"a":"559B84E8B975","b":"559B83A79000","o":"1412975","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"F5"},\{"a":"559B84E89685","b":"559B83A79000","o":"1410685","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"},\{"a":"559B84E8A616","b":"559B83A79000","o":"1411616","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B86148162","b":"559B83A79000","o":"26CF162","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"182"},\{"a":"559B84E88A9B","b":"559B83A79000","o":"140FA9B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"},\{"a":"559B84E89DFD","b":"559B83A79000","o":"1410DFD","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"6AD"},\{"a":"559B84E89EE0","b":"559B83A79000","o":"1410EE0","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"},\{"a":"559B84E8A185","b":"559B83A79000","o":"1411185","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"},\{"a":"559B84E896FA","b":"559B83A79000","o":"14106FA","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"},\{"a":"559B84E8A616","b":"559B83A79000","o":"1411616","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B861487C8","b":"559B83A79000","o":"26CF7C8","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"B8"},\{"a":"559B864991E6","b":"559B83A79000","o":"2A201E6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"559B86499254","b":"559B83A79000","o":"2A20254","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"},\{"a":"7F336A2E9EA5","b":"7F336A2E2000","o":"7EA5","s":"start_thread","s+":"C5"},\{"a":"7F336A0129FD","b":"7F3369F14000","o":"FE9FD","s":"clone","s+":"6D"}],"processInfo":\{"mongodbVersion":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","compiledModules":[],"uname":{"sysname":"Linux","release":"3.10.0-1160.15.2.el7.x86_64","version":"#1 SMP Wed Feb 3 15:06:38 UTC 2021","machine":"x86_64"},"somap":[\{"b":"559B83A79000","elfType":3,"buildId":"CC93F86C6D00CE717DB23676747D258B4027E066"},\{"b":"7F336A2E2000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B"},\{"b":"7F3369F14000","path":"/lib64/libc.so.6","elfType":3,"buildId":"7CD4A08C18C60E1E2EA1AEBB88C9379BD7289D38"}]}}}} 
      

      I tried also with the version 4.4.3 but have the same errors.

       

        1. mongod.log.rpmsave.gz
          145 kB
        2. mongod.log.rpmsave.zip
          145 kB

            Assignee:
            backlog-server-repl [DO NOT USE] Backlog - Replication Team
            Reporter:
            nicola.battista89@gmail.com Nicola Battista
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved:
              None
              None
              None
              None