Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63990

Segmentation fault on primary

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • ALL

      We have a MongoDB cluster setup with following configuration

      • 1 primary
      • 2 secondaries (1 connected to clients but not used for reads, other one not connected to clients at all)
      • 1 arbiter

      Throughput pattern is write heavy with comparatively lesser number of reads.

      A few days back, our primary crashed out of nowhere. It emitted the following traceback,

      2022-02-21T02:45:58.828+0000 F -        [thread12203756] Invalid access at address: 0xe8
      2022-02-21T02:45:58.863+0000 F -        [thread12203756] Got signal: 11 (Segmentation fault).
      
       0x562f500b44f1 0x562f500b3709 0x562f500b3d76 0x7fca6ebb9390 0x562f4e9abf0f 0x562f4e9aee8b 0x562f4e939294 0x562f4e93a2e2 0x562f4e9332b3 0x562f4e933647 0x562f4e935a22 0x562f4e97ec79 0x7fca6ebaf6ba 0x7fca6e8e551d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"562F4DEC1000","o":"21F34F1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562F4DEC1000","o":"21F2709"},{"b":"562F4DEC1000","o":"21F2D76"},{"b":"7FCA6EBA8000","o":"11390"},{"b":"562F4DEC1000","o":"AEAF0F"},{"b":"562F4DEC1000","o":"AEDE8B","s":"__wt_split_reverse"},{"b":"562F4DEC1000","o":"A78294"},{"b":"562F4DEC1000","o":"A792E2","s":"__wt_evict"},{"b":"562F4DEC1000","o":"A722B3"},{"b":"562F4DEC1000","o":"A72647"},{"b":"562F4DEC1000","o":"A74A22","s":"__wt_evict_thread_run"},{"b":"562F4DEC1000","o":"ABDC79"},{"b":"7FCA6EBA8000","o":"76BA"},{"b":"7FCA6E7DE000","o":"10751D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.2", "gitVersion" : "489d177dbd0f0420a8ca04d39fd78d0a2c539420", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-1109-aws", "version" : "#120-Ubuntu SMP Fri Jun 5 01:26:57 UTC 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "562F4DEC1000", "elfType" : 3, "buildId" : "90F4CC751C09ABD90756CE2480F0217355B846B5" }, { "b" : "7FFC77559000", "elfType" : 3, "buildId" : "E84E25DBB968253D84247A3C45F8B83709552AC9" }, { "b" : "7FCA6FD9D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C8ACB5535F4B7B9880A62B4B38EBE924668F6446" }, { "b" : "7FCA6FB35000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "069C06FA5B5598CB6B27089BD4BEC60469AC85F7" }, { "b" : "7FCA6F6F0000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3662558D7453C25E2ABE7E6B49D9256F1B8BB1A7" }, { "b" : "7FCA6F4E8000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "BB404D52807964CCC7F0815BC2666688A74B958F" }, { "b" : "7FCA6F2E4000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "BBA6A2E958188C44B9BDA990278EBE8868B85379" }, { "b" : "7FCA6EFDB000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "E178A25E6DB28598588C03D898E44FD79BD16E4D" }, { "b" : "7FCA6EDC5000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FCA6EBA8000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C557B8146E8079AF46310B549DE6912D1FC4EA86" }, { "b" : "7FCA6E7DE000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30773BE8CF5BFED9D910C8473DD44EAAB2E705AB" }, { "b" : "7FCA6FFB8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "98D7BC4313D0D8D5E127E06ACF2319829C5CE61D" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x562f500b44f1]
       mongod(+0x21F2709) [0x562f500b3709]
       mongod(+0x21F2D76) [0x562f500b3d76]
       libpthread.so.0(+0x11390) [0x7fca6ebb9390]
       mongod(+0xAEAF0F) [0x562f4e9abf0f]
       mongod(__wt_split_reverse+0x11B) [0x562f4e9aee8b]
       mongod(+0xA78294) [0x562f4e939294]
       mongod(__wt_evict+0xFE2) [0x562f4e93a2e2]
       mongod(+0xA722B3) [0x562f4e9332b3]
       mongod(+0xA72647) [0x562f4e933647]
       mongod(__wt_evict_thread_run+0x72) [0x562f4e935a22]
       mongod(+0xABDC79) [0x562f4e97ec79]
       libpthread.so.0(+0x76BA) [0x7fca6ebaf6ba]
       libc.so.6(clone+0x6D) [0x7fca6e8e551d]
      -----  END BACKTRACE  -----

            Assignee:
            edwin.zhou@mongodb.com Edwin Zhou
            Reporter:
            tejas.jadhav@gojek.com Tejas Jadhav
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: