Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-53708

Excess memory usage during shutdown in durable history tests

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • ALL
    • v5.1
    • Hide

      Use DSI locally to setup a perf cluster:

      1. On the iwc node with the following command line: 
         (cd src/genny && ./src/workloads/contrib/historystore/eMRCf_runner.sh -g 8 -n)

       

      Show
      Use DSI locally to setup a perf cluster: On the iwc node with the following command line: (cd src/genny && ./src/workloads/contrib/historystore/eMRCf_runner.sh -g 8 -n)  
    • Storage - Ra 2021-09-20, Storage - Ra 2021-10-04, Storage - Ra 2021-10-18
    • 3

      Executing the eMRCf_runner.sh tests with more than 7 growth iterations and enableMajorityReadConcern set to true results in a SIGAbort when shutting down the primary.

      {"t":{"$date":"2021-01-10T00:14:48.681+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:672274][43957:0x7ff5de9fd700], file:index-39-3882910392635076206.wt, txn rollback_to_stable: __wt_calloc, 52: memory allocation of 2848 bytes failed: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:48.695+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:695405][43957:0x7ff5de9fd700], WT_CONNECTION.close: __wt_calloc, 52: memory allocation of 264400 bytes failed: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:48.697+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:696255][43957:0x7ff5de9fd700], WT_CONNECTION.close: __conn_close, 1035: failure during close, disabling further writes: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:49.054+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread355","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1610237689:54366][43957:0x7ff60225f700], log-wrlsn-server: __log_wrlsn_server, 871: log wrlsn server error: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2021-01-10T00:14:49.057+00:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"thread355","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":537}}
{"t":{"$date":"2021-01-10T00:14:49.057+00:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"thread355","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2021-01-10T00:14:49.059+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"thread355","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-01-10T00:14:49.238+00:00"},"s":"E",  "c":"CONTROL",  "id":31430,   "ctx":"thread355","msg":"Error collecting stack trace","attr":{"error":"unw_get_proc_name(564559009DFA): unspecified (general) error\nunw_get_proc_name(56455900B1F9): unspecified (general) error\nunw_get_proc_name(564559006DF6): unspecified (general) error\nunw_get_proc_name(7FF60AE1A7E0): unspecified (general) error\nunw_get_proc_name(7FF60AA91B20): unspecified (general) error\nunw_get_proc_name(7FF60AA92FC8): unspecified (general) error\nunw_get_proc_name(5645568CCE5A): unspecified (general) error\nunw_get_proc_name(5645565CA91F): unspecified (general) error\nunw_get_proc_name(5645573CF68B): unspecified (general) error\nunw_get_proc_name(5645565E81BE): unspecified (general) error\nunw_get_proc_name(5645565D376B): unspecified (general) error\nunw_get_proc_name(7FF60AE1040B): unspecified (general) error\nunw_get_proc_name(7FF60AB4AF9F): unspecified (general) error\nunw_get_proc_name(564559009DFA): unspecified (general) error\nunw_get_proc_name(56455900B1F9): unspecified (general) error\nunw_get_proc_name(564559006DF6): unspecified (general) error\nunw_get_proc_name(7FF60AE1A7E0): unspecified (general) error\nunw_get_proc_name(7FF60AA91B20): unspecified (general) error\nunw_get_proc_name(7FF60AA92FC8): unspecified (general) error\nunw_get_proc_name(5645568CCE5A): unspecified (general) error\nunw_get_proc_name(5645565CA91F): unspecified (general) error\nunw_get_proc_name(5645573CF68B): unspecified (general) error\nunw_get_proc_name(5645565E81BE): unspecified (general) error\nunw_get_proc_name(5645565D376B): unspecified (general) error\nunw_get_proc_name(7FF60AE1040B): unspecified (general) error\nunw_get_proc_name(7FF60AB4AF9F): unspecified (general) error\n"}}
{"t":{"$date":"2021-01-10T00:14:49.238+00:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"thread355","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"564559009DFA","b":"56455568F000","o":"397ADFA"},{"a":"56455900B1F9","b":"56455568F000","o":"397C1F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"564559006DF6","b":"56455568F000","o":"3977DF6","s":"abruptQuit","s+":"66"},{"a":"7FF60AE1A7E0","b":"7FF60AE09000","o":"117E0"},{"a":"7FF60AA91B20","b":"7FF60AA5E000","o":"33B20","s":"gsignal","s+":"110"},{"a":"7FF60AA92FC8","b":"7FF60AA5E000","o":"34FC8","s":"abort","s+":"148"},{"a":"5645568CCE5A","b":"56455568F000","o":"123DE5A","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"0"},{"a":"5645565CA91F","b":"56455568F000","o":"F3B91F"},{"a":"5645573CF68B","b":"56455568F000","o":"1D4068B"},{"a":"5645565E81BE","b":"56455568F000","o":"F591BE","s":"__wt_panic_func","s+":"B3"},{"a":"5645565D376B","b":"56455568F000","o":"F4476B"},{"a":"7FF60AE1040B","b":"7FF60AE09000","o":"740B"},{"a":"7FF60AB4AF9F","b":"7FF60AA5E000","o":"ECF9F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.9.0-alpha-1214-g54e8728","gitVersion":"54e87285f802a49456c02c80cb0ddb0fbb54c88a","compiledModules":["enterprise"],"uname":{"sysname":"Linux","release":"4.14.165-131.185.amzn2.x86_64","version":"#1 SMP Wed Jan 15 14:19:56 UTC 2020","machine":"x86_64"},"somap":[{"b":"56455568F000","elfType":3,"buildId":"7FEA37A074A30C9B75AC461167B7598F4E171408"},{"b":"7FF60AE09000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"5E12AD1B2983169B48CE4C7051454E8CA38D6EF0"},{"b":"7FF60AA5E000","path":"/lib64/libc.so.6","elfType":3,"buildId":"DAFC6796B1B45E202959E8872844CABEE8A20A32"}]}}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"564559009DFA","b":"56455568F000","o":"397ADFA"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"56455900B1F9","b":"56455568F000","o":"397C1F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"564559006DF6","b":"56455568F000","o":"3977DF6","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AE1A7E0","b":"7FF60AE09000","o":"117E0"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AA91B20","b":"7FF60AA5E000","o":"33B20","s":"gsignal","s+":"110"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AA92FC8","b":"7FF60AA5E000","o":"34FC8","s":"abort","s+":"148"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645568CCE5A","b":"56455568F000","o":"123DE5A","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"0"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565CA91F","b":"56455568F000","o":"F3B91F"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645573CF68B","b":"56455568F000","o":"1D4068B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565E81BE","b":"56455568F000","o":"F591BE","s":"__wt_panic_func","s+":"B3"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565D376B","b":"56455568F000","o":"F4476B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AE1040B","b":"7FF60AE09000","o":"740B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AB4AF9F","b":"7FF60AA5E000","o":"ECF9F","s":"clone","s+":"3F"}}}

        1. mem.png
          mem.png
          133 kB
        2. Screen Shot 2021-10-13 at 4.11.49 pm.png
          Screen Shot 2021-10-13 at 4.11.49 pm.png
          127 kB
        3. Screen Shot 2021-10-13 at 4.12.46 pm.png
          Screen Shot 2021-10-13 at 4.12.46 pm.png
          34 kB
        4. Screen Shot 2021-10-13 at 5.22.32 pm.png
          Screen Shot 2021-10-13 at 5.22.32 pm.png
          43 kB

            Assignee:
            chenhao.qu@mongodb.com Chenhao Qu
            Reporter:
            jim.oleary@mongodb.com James O'Leary
            Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: