Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38292

mongodb crash with Got signal: 11 (Segmentation fault)

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL
    • Storage Engines 2019-02-25, Storage Engines 2019-06-03
    • 0

      We are also having this issue both on 3.2.19 and 3.4.14 (both are WT engine). Here's the stack from 3.4.14 logs. Can you brief me on what exactly is the fix in WT-4037. It is not clear in that ticket.

      2018-11-28T10:06:13.885-0600 F - [thread2] Invalid access at address: 0
      2018-11-28T10:06:13.906-0600 F - [thread2] Got signal: 11 (Segmentation fault). 0x562636fa4a51 0x562636fa3c69 0x562636fa42d6 0x7f74ee9647e0 0x56263716bca3 0x56263716bd7c 0x562637aa1d7a 0x5626378e0466 0x5626378e05ea 0x5626378e0c08 0x5626378e3ac3 0x562637934294 0x562637934e47 0x56263792f813 0x56263792fba7 0x5626379316a3 0x56263799b906 0x7f74ee95caa1 0x7f74ee6a9c4d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[\{"b":"562635A2B000","o":"1579A51","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"562635A2B000","o":"1578C69"},\{"b":"562635A2B000","o":"15792D6"},\{"b":"7F74EE955000","o":"F7E0"},\{"b":"562635A2B000","o":"1740CA3","s":"_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi"},\{"b":"562635A2B000","o":"1740D7C","s":"_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm"},\{"b":"562635A2B000","o":"2076D7A","s":"_ZdlPvRKSt9nothrow_t"},\{"b":"562635A2B000","o":"1EB5466","s":"__wt_split_stash_discard"},\{"b":"562635A2B000","o":"1EB55EA"},\{"b":"562635A2B000","o":"1EB5C08"},\{"b":"562635A2B000","o":"1EB8AC3","s":"__wt_split_reverse"},\{"b":"562635A2B000","o":"1F09294"},\{"b":"562635A2B000","o":"1F09E47","s":"__wt_evict"},\{"b":"562635A2B000","o":"1F04813"},\{"b":"562635A2B000","o":"1F04BA7"},\{"b":"562635A2B000","o":"1F066A3","s":"__wt_evict_thread_run"},\{"b":"562635A2B000","o":"1F70906","s":"__wt_thread_run"},\{"b":"7F74EE955000","o":"7AA1"},\{"b":"7F74EE5C1000","o":"E8C4D","s":"clone"}],"processInfo":\{ "mongodbVersion" : "3.4.14", "gitVersion" : "fd954412dfc10e4d1e3e2dd4fac040f8b476b268", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.6.3.el6.x86_64", "version" : "#1 SMP Tue Oct 9 17:27:49 UTC 2018", "machine" : "x86_64" }, "somap" : [ \{ "b" : "562635A2B000", "elfType" : 3, "buildId" : "64CD384C41ACC8D81741F0DFC0F9A3D7756F81FF" }, \{ "b" : "7FFF3E8F6000", "elfType" : 3, "buildId" : "F9F48CC73D4D61AE273899B31855C6589EE5EA8D" }, \{ "b" : "7F74EF7FD000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BECFB85A8BC084042D5BF2BA9E66325CE798B659" }, \{ "b" : "7F74EF418000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "CBDA444A7109874C5350AE9CEEF3F82F749B347F" }, \{ "b" : "7F74EF210000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "552CEC3216281CCFD7FA6432C723D50163255823" }, \{ "b" : "7F74EF00C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "2AF795BFFD122309BA3359FEBABB5D0967403D17" }, \{ "b" : "7F74EED88000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4AAEE970B045D8BF946578B9C7F3AB5CDE9AB44A" }, \{ "b" : "7F74EEB72000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, \{ "b" : "7F74EE955000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4EA475CD3FD3B69B6C95D9381FA74B36DB4992EF" }, \{ "b" : "7F74EE5C1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "BCA7789C2EA8E28CB7CE553E183AC7E7EE36F8A2" }, \{ "b" : "7F74EFA69000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, \{ "b" : "7F74EE37D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, \{ "b" : "7F74EE096000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, \{ "b" : "7F74EDE92000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, \{ "b" : "7F74EDC66000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, \{ "b" : "7F74EDA50000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, \{ "b" : "7F74ED845000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, \{ "b" : "7F74ED642000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, \{ "b" : "7F74ED428000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4786A2A5D30B121601958E84D643C70C13C4FBA5" }, \{ "b" : "7F74ED209000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
       mongod-3.4(_ZN5mongo15printStackTraceERSo+0x41) [0x562636fa4a51]
       mongod-3.4(+0x1578C69) [0x562636fa3c69]
       mongod-3.4(+0x15792D6) [0x562636fa42d6]
       libpthread.so.0(+0xF7E0) [0x7f74ee9647e0]
       mongod-3.4(_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi+0xE3) [0x56263716bca3]
       mongod-3.4(_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm+0x1C) [0x56263716bd7c]
       mongod-3.4(_ZdlPvRKSt9nothrow_t+0x26A) [0x562637aa1d7a]
       mongod-3.4(__wt_split_stash_discard+0xC6) [0x5626378e0466]
       mongod-3.4(+0x1EB55EA) [0x5626378e05ea]
       mongod-3.4(+0x1EB5C08) [0x5626378e0c08]
       mongod-3.4(__wt_split_reverse+0x83) [0x5626378e3ac3]
       mongod-3.4(+0x1F09294) [0x562637934294]
       mongod-3.4(__wt_evict+0xAC7) [0x562637934e47]
       mongod-3.4(+0x1F04813) [0x56263792f813]
       mongod-3.4(+0x1F04BA7) [0x56263792fba7]
       mongod-3.4(__wt_evict_thread_run+0xD3) [0x5626379316a3]
       mongod-3.4(__wt_thread_run+0x16) [0x56263799b906]
       libpthread.so.0(+0x7AA1) [0x7f74ee95caa1]
       libc.so.6(clone+0x6D) [0x7f74ee6a9c4d]
      ----- END BACKTRACE -----
      

       

      Please note that we are in the process of generating a core dump and will attach here when the mongod process crashes again.

        1. local_dmesg.txt
          225 kB
        2. local_system_log_Nov28.log
          9 kB

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            manan@indeed.com Manan Shah
            Votes:
            1 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: