Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33057

Indexing fatal assertion on secondary during sync

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.4.10
    • None
    • Environment:
      Linux

      We recently added a new hidden secondary to one of our many clusters. The cluster, while on version 3.4.10, is running in featureCompatabilityMode: 3.2. After initial sync, while rebuilding numerous indexes, the server crashed while attempting to create what appears to have been its 65th index.

      I suspect a fatal exception here is not ideal. I'm also curious how our other systems are still alive, while on the same version.

      2018-02-01T11:19:12.226-0600 I INDEX    [repl index builder 1134] build index on: XDemo2.D_22 properties: { v: 1, key: { Fx.A.SV: 1 }, name: "Fx.A.SV_1", ns: "XDemo2.D_22", background: true }
      2018-02-01T11:19:12.253-0600 I INDEX    [repl index builder 1134] build index done.  scanned 75 total records. 0 secs
      2018-02-01T11:19:12.254-0600 I INDEX    [repl index builder 1135] add index fails, too many indexes for XDemo2.D_22 key:{ Fx.A.RRIds: 1 }
      2018-02-01T11:19:12.254-0600 E INDEX    [repl index builder 1135] bad status from index build: CannotCreateIndex: add index fails, too many indexes for XDemo2.D_22 key:{ Fx.A.RRIds: 1 }
      2018-02-01T11:19:12.255-0600 E INDEX    [repl index builder 1135] IndexBuilder could not build index: CannotCreateIndex: add index fails, too many indexes for XDemo2.D_22 key:{ Fx.A.RRIds: 1 }
      2018-02-01T11:19:12.255-0600 I -        [repl index builder 1135] Fatal Assertion 28555 at src/mongo/db/index_builder.cpp 106
      2018-02-01T11:19:12.255-0600 I -        [repl index builder 1135]
      
      ***aborting after fassert() failure
      
      
      2018-02-01T11:19:12.327-0600 F -        [repl index builder 1135] Got signal: 6 (Aborted).
      
       0x558738e68641 0x558738e67859 0x558738e67d3d 0x7f26d476a890 0x7f26d43e5067 0x7f26d43e6448 0x5587381127a3 0x558738576be6 0x558738ddac41 0x5587398dda40 0x7f26d4763064 0x7f26d449862d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"5587378F7000","o":"1571641","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5587378F7000","o":"1570859"},{"b":"5587378F7000","o":"1570D3D"},{"b":"7F26D475B000","o":"F890"},{"b":"7F26D43B0000","o":"35067","s":"gsignal"},{"b":"7F26D43B0000","o":"36448","s":"abort"},{"b":"5587378F7000","o":"81B7A3","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5587378F7000","o":"C7FBE6","s":"_ZN5mongo12IndexBuilder3runEv"},{"b":"5587378F7000","o":"14E3C41","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"5587378F7000","o":"1FE6A40"},{"b":"7F26D475B000","o":"8064"},{"b":"7F26D43B0000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-4-amd64", "version" : "#1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19)", "machine" : "x86_64" }, "somap" : [ { "b" : "5587378F7000", "elfType" : 3, "buildId" : "004E9FAB8E542F8BABDB0274A211B1E9D15DEDF1" }, { "b" : "7FFE359F5000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "D3D8F25E67DC1A88F1545D51C29476B4CC91F2E5" }, { "b" : "7F26D5697000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "21115992A1F885E1ACE88AADA60F126AD9759D03" }, { "b" : "7F26D529B000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "FD6376149047833953B0269E84DE181CA45DBE90" }, { "b" : "7F26D5093000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7F26D4E8F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7F26D4B8E000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7F26D4978000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F26D475B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7F26D43B0000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7F26D58F8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x558738e68641]
       mongod(+0x1570859) [0x558738e67859]
       mongod(+0x1570D3D) [0x558738e67d3d]
       libpthread.so.0(+0xF890) [0x7f26d476a890]
       libc.so.6(gsignal+0x37) [0x7f26d43e5067]
       libc.so.6(abort+0x148) [0x7f26d43e6448]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5587381127a3]
       mongod(_ZN5mongo12IndexBuilder3runEv+0xD86) [0x558738576be6]
       mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x558738ddac41]
       mongod(+0x1FE6A40) [0x5587398dda40]
       libpthread.so.0(+0x8064) [0x7f26d4763064]
       libc.so.6(clone+0x6D) [0x7f26d449862d]
      -----  END BACKTRACE  -----
      
      

            Assignee:
            william.schultz@mongodb.com Will Schultz
            Reporter:
            sallgeud Chad Kreimendahl
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: