Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52735

mongodb crash with "Invariant failure" opCtx != nullptr && _opCtx == nullptr

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.0
    • Component/s: Stability
    • Labels:
    • Service Arch
    • ALL
    • Security 2020-11-30

      Running mondodb 4.4.0 on Ubuntu 20.04.1 one of our replicas crashed with an assertion. How do we diagnose this?

       

      {"t":{"$date":"2020-11-10T14:07:13.371+00:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"monitoring-keys-for-HMAC","msg":"Invariant failure","attr":{"expr":"opCtx != nullptr && _opCtx == nullptr","file":"src/mongo/db/client.cpp","line":126}}
{"t":{"$date":"2020-11-10T14:07:13.371+00:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"monitoring-keys-for-HMAC","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2020-11-10T14:07:13.373+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"monitoring-keys-for-HMAC","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"monitoring-keys-for-HMAC","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"556342B47811","b":"55633FE8C000","o":"2CBB811","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},{"a":"556342B48EB9","b":"55633FE8C000","o":"2CBCEB9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"556342B466A6","b":"55633FE8C000","o":"2CBA6A6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7F8DEAD7C3C0","b":"7F8DEAD67000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"7F8DEABBB18B","b":"7F8DEAB75000","o":"4618B","s":"gsignal","s+":"CB"},{"a":"7F8DEAB9A859","b":"7F8DEAB75000","o":"25859","s":"abort","s+":"12B"},{"a":"556340D8FBE1","b":"55633FE8C000","o":"F03BE1","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"},{"a":"556340D65270","b":"55633FE8C000","o":"ED9270","s":"_ZN5mongo6Client19setOperationContextEPNS_16OperationContextE.cold.135","s+":"18"},{"a":"556342A15BE9","b":"55633FE8C000","o":"2B89BE9","s":"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE","s+":"129"},{"a":"556342A0A697","b":"55633FE8C000","o":"2B7E697","s":"_ZN5mongo6Client20makeOperationContextEv","s+":"27"},{"a":"556342809712","b":"55633FE8C000","o":"297D712","s":"_ZN5mongo21KeysCollectionManager14PeriodicRunner18_doPeriodicRefreshEPNS_14ServiceContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8DurationISt5ratioILl1ELl1000EEEE","s+":"162"},{"a":"55634280B6C3","b":"55633FE8C000","o":"297F6C3","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_21KeysCollectionManager14PeriodicRunner5startEPNS3_14ServiceContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS3_8DurationISt5ratioILl1ELl1000EEEEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"93"},{"a":"556342CF78FF","b":"55633FE8C000","o":"2E6B8FF","s":"execute_native_thread_routine","s+":"F"},{"a":"7F8DEAD70609","b":"7F8DEAD67000","o":"9609","s":"start_thread","s+":"D9"},{"a":"7F8DEAC97293","b":"7F8DEAB75000","o":"122293","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-1025-aws","version":"#25-Ubuntu SMP Fri Sep 11 09:37:24 UTC 2020","machine":"x86_64"},"somap":[{"b":"55633FE8C000","elfType":3,"buildId":"77B6A138746C90015067F12B963853BD51DAA5A6"},{"b":"7F8DEAD67000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"4FC5FC33F4429136A494C640B113D76F610E4ABC"},{"b":"7F8DEAB75000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"F3FF3FDA80B817C464A56EED59FF09DC864EAEB0"}]}}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342B47811","b":"55633FE8C000","o":"2CBB811","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342B48EB9","b":"55633FE8C000","o":"2CBCEB9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342B466A6","b":"55633FE8C000","o":"2CBA6A6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F8DEAD7C3C0","b":"7F8DEAD67000","o":"153C0","s":"funlockfile","s+":"60"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F8DEABBB18B","b":"7F8DEAB75000","o":"4618B","s":"gsignal","s+":"CB"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F8DEAB9A859","b":"7F8DEAB75000","o":"25859","s":"abort","s+":"12B"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556340D8FBE1","b":"55633FE8C000","o":"F03BE1","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556340D65270","b":"55633FE8C000","o":"ED9270","s":"_ZN5mongo6Client19setOperationContextEPNS_16OperationContextE.cold.135","s+":"18"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342A15BE9","b":"55633FE8C000","o":"2B89BE9","s":"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE","s+":"129"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342A0A697","b":"55633FE8C000","o":"2B7E697","s":"_ZN5mongo6Client20makeOperationContextEv","s+":"27"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342809712","b":"55633FE8C000","o":"297D712","s":"_ZN5mongo21KeysCollectionManager14PeriodicRunner18_doPeriodicRefreshEPNS_14ServiceContextENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_8DurationISt5ratioILl1ELl1000EEEE","s+":"162"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"55634280B6C3","b":"55633FE8C000","o":"297F6C3","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_21KeysCollectionManager14PeriodicRunner5startEPNS3_14ServiceContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS3_8DurationISt5ratioILl1ELl1000EEEEEUlvE_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"93"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"556342CF78FF","b":"55633FE8C000","o":"2E6B8FF","s":"execute_native_thread_routine","s+":"F"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F8DEAD70609","b":"7F8DEAD67000","o":"9609","s":"start_thread","s+":"D9"}}}
{"t":{"$date":"2020-11-10T14:07:13.480+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"monitoring-keys-for-HMAC","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F8DEAC97293","b":"7F8DEAB75000","o":"122293","s":"clone","s+":"43"}}}{"t":{"$date":"2020-11-10T14:13:25.412+00:00"},"s":"W",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****","tags":["startupWarnings"]}

        1. Screenshot from 2021-02-16 14-59-15.png
          Screenshot from 2021-02-16 14-59-15.png
          35 kB

            Votes:
            10 Vote for this issue
            Watchers:
            41 Start watching this issue

              Created:
              Updated:
              Resolved: