Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16886

Segmentation fault when loading data into WiredTiger.

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.0-rc6
    • Affects Version/s: 2.8.0-rc5
    • Component/s: Storage
    • Environment:
      Amazon Linux.
    • Fully Compatible
    • ALL
    • Hide

      I can give you access to the servers , I am of course based in Europe so w need to work out timing.

      Show
      I can give you access to the servers , I am of course based in Europe so w need to work out timing.

      Loading data on a high spec server on EC2 + EBS
      15 x 400GB EBS SSD Diks as Raid 0.

      Using custom C based multi threaded loader to load 500,000 real records (~270GB).

      Wired tiger segfaults, seemingly at different points but reproducibly.

      Stack trace.

      2015-01-15T22:40:21.108+0000 F -        [conn3] Invalid access at address: 0x20
      2015-01-15T22:40:21.119+0000 F -        [conn3] Got signal: 11 (Segmentation fault).
      
       0xf25749 0xf24df2 0xf2511e 0x7faab058d5b0 0x12d11d3 0x13092ce 0x1309896 0x130756d 0x12c62cd 0x12df145 0x12ba500 0x12f4eb6 0xd47aed 0x8f073e 0x8f0f0c 0xa8f868 0xa8fce0 0xa92442 0xa94d48 0x7e5320 0xee37d1 0x7faab0585f18 0x7faaaf697b9d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B25749"},{"b":"400000","o":"B24DF2"},{"b":"400000","o":"B2511E"},{"b":"7FAAB057E000","o":"F5B0"},{"b":"400000","o":"ED11D3"},{"b":"400000","o":"F092CE"},{"b":"400000","o":"F09896"},{"b":"400000","o":"F0756D"},{"b":"400000","o":"EC62CD"},{"b":"400000","o":"EDF145"},{"b":"400000","o":"EBA500"},{"b":"400000","o":"EF4EB6"},{"b":"400000","o":"947AED"},{"b":"400000","o":"4F073E"},{"b":"400000","o":"4F0F0C"},{"b":"400000","o":"68F868"},{"b":"400000","o":"68FCE0"},{"b":"400000","o":"692442"},{"b":"400000","o":"694D48"},{"b":"400000","o":"3E5320"},{"b":"400000","o":"AE37D1"},{"b":"7FAAB057E000","o":"7F18"},{"b":"7FAAAF5B5000","o":"E2B9D"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc5", "gitVersion" : "74b351de21c84438b12a83b28e155f5e69e3c1eb", "uname" : { "sysname" : "Linux", "release" : "3.14.20-20.44.amzn1.x86_64", "version" : "#1 SMP Mon Oct 6 22:52:46 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFB9AFE000", "elfType" : 3 }, { "b" : "7FAAB057E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3 }, { "b" : "7FAAB0376000", "path" : "/lib64/librt.so.1", "elfType" : 3 }, { "b" : "7FAAB0172000", "path" : "/lib64/libdl.so.2", "elfType" : 3 }, { "b" : "7FAAAFE6E000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FAAAFB70000", "path" : "/lib64/libm.so.6", "elfType" : 3 }, { "b" : "7FAAAF95A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FAAAF5B5000", "path" : "/lib64/libc.so.6", "elfType" : 3 }, { "b" : "7FAAB079A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf25749]
       mongod(+0xB24DF2) [0xf24df2]
       mongod(+0xB2511E) [0xf2511e]
       libpthread.so.0(+0xF5B0) [0x7faab058d5b0]
       mongod(__wt_split_insert+0x813) [0x12d11d3]
       mongod(+0xF092CE) [0x13092ce]
       mongod(__wt_evict+0x96) [0x1309896]
       mongod(__wt_evict_page+0x2D) [0x130756d]
       mongod(__wt_page_in_func+0x64D) [0x12c62cd]
       mongod(__wt_row_search+0x9D5) [0x12df145]
       mongod(__wt_btcur_insert+0x300) [0x12ba500]
       mongod(+0xEF4EB6) [0x12f4eb6]
       mongod(_ZN5mongo21WiredTigerRecordStore12insertRecordEPNS_16OperationContextEPKcib+0xCD) [0xd47aed]
       mongod(_ZN5mongo10Collection15_insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x4E) [0x8f073e]
       mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEb+0x8C) [0x8f0f0c]
       mongod(_ZN5mongo14checkAndInsertEPNS_16OperationContextERNS_6Client7ContextEPKcRNS_7BSONObjE+0x108) [0xa8f868]
       mongod(_ZN5mongo11insertMultiEPNS_16OperationContextERNS_6Client7ContextEbPKcRSt6vectorINS_7BSONObjESaIS8_EERNS_5CurOpE+0xB0) [0xa8fce0]
       mongod(_ZN5mongo14receivedInsertEPNS_16OperationContextERNS_7MessageERNS_5CurOpE+0x522) [0xa92442]
       mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0x1658) [0xa94d48]
       mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7e5320]
       mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x321) [0xee37d1]
       libpthread.so.0(+0x7F18) [0x7faab0585f18]
       libc.so.6(clone+0x6D) [0x7faaaf697b9d]
      -----  END BACKTRACE  -----
      
      

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            john.page@mongodb.com John Page
            Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: