Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-28241

OOM error while adding a member to replica set

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None
    • Operating System:
      ALL

      Description

      Hi,
      I am trying to add a new memory to my replica set. It goes on smoothly for several hours, when I get the following error:

      2017-03-08T02:53:42.744+0000 I REPL     [repl writer worker 2] CollectionCloner::start called, on ns:8ed39e1c988ff713bc13.events
      2017-03-08T02:53:42.751+0000 I INDEX    [InitialSyncInserters-8ed39e1c988ff713bc13.events0] build index on: 8ed39e1c988ff713bc13.events properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "8ed39e1c988ff713bc13.events" }
      2017-03-08T02:53:42.751+0000 I INDEX    [InitialSyncInserters-8ed39e1c988ff713bc13.events0]      building index using bulk method; build may temporarily use up to 500 megabytes of RAM
      2017-03-08T02:54:36.840+0000 F -        [InitialSyncInserters-8ed39e1c988ff713bc13.events0] out of memory.
       
       0x7f9bb0880221 0x7f9bb087f734 0x7f9bb0a3d2cb 0x7f9bb1377072 0x7f9baff76cb5 0x7f9baff76d52 0x7f9baff7df8a 0x7f9baff7e61d 0x7f9baff6d749 0x7f9bafd2076b 0x7f9bb0218a91 0x7f9bb0219673 0x7f9bb02196f5 0x7f9bb0369341 0x7f9bb0369577 0x7f9bb036a75f 0x7f9bb07f72c5 0x7f9bb07f7df0 0x7f9bb07f8999 0x7f9bb12f51d0 0x7f9badf63184 0x7f9badc9037d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"7F9BAF301000","o":"157F221","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F9BAF301000","o":"157E734","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"7F9BAF301000","o":"173C2CB"},{"b":"7F9BAF301000","o":"2076072","s":"tc_newarray_nothrow"},{"b":"7F9BAF301000","o":"C75CB5","s":"_ZSt20get_temporary_bufferISt4pairIN5mongo7BSONObjENS1_8RecordIdEEES0_IPT_lEl"},{"b":"7F9BAF301000","o":"C75D52","s":"_ZNSt17_Temporary_bufferISt15_Deque_iteratorISt4pairIN5mongo7BSONObjENS2_8RecordIdEERS5_PS5_ES5_EC1ES8_S8_"},{"b":"7F9BAF301000","o":"C7CF8A","s":"_ZN5mongo6sorter13NoLimitSorterINS_7BSONObjENS_8RecordIdENS_27BtreeExternalSortComparisonEE4sortEv"},{"b":"7F9BAF301000","o":"C7D61D","s":"_ZN5mongo6sorter13NoLimitSorterINS_7BSONObjENS_8RecordIdENS_27BtreeExternalSortComparisonEE4doneEv"},{"b":"7F9BAF301000","o":"C6C749","s":"_ZN5mongo17IndexAccessMethod10commitBulkEPNS_16OperationContextESt10unique_ptrINS0_11BulkBuilderESt14default_deleteIS4_EEbbPSt3setINS_8RecordIdESt4lessIS9_ESaIS9_EE"},{"b":"7F9BAF301000","o":"A1F76B","s":"_ZN5mongo15MultiIndexBlock13doneInsertingEPSt3setINS_8RecordIdESt4lessIS2_ESaIS2_EE"},{"b":"7F9BAF301000","o":"F17A91"},{"b":"7F9BAF301000","o":"F18673"},{"b":"7F9BAF301000","o":"F186F5"},{"b":"7F9BAF301000","o":"1068341"},{"b":"7F9BAF301000","o":"1068577"},{"b":"7F9BAF301000","o":"106975F","s":"_ZN5mongo4repl10TaskRunner9_runTasksEv"},{"b":"7F9BAF301000","o":"14F62C5","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"7F9BAF301000","o":"14F6DF0","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"7F9BAF301000","o":"14F7999","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"7F9BAF301000","o":"1FF41D0"},{"b":"7F9BADF5B000","o":"8184"},{"b":"7F9BADB96000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.1", "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-105-generic", "version" : "#152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "7F9BAF301000", "elfType" : 3, "buildId" : "B0B5767DC6B7AB9D129191D0FA1F55666B34170F" }, { "b" : "7FFCE42F5000", "elfType" : 3, "buildId" : "9C7CBCF6C957D8FC8E55B45A3C7A1556B38A3097" }, { "b" : "7F9BAEE7D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "48A664AE6B0B4918A3EB0156C6364C4F084232FD" }, { "b" : "7F9BAEAA1000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "6B8997EA892A7FF37AC8CAA8F239D595251889BB" }, { "b" : "7F9BAE899000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F9BAE695000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F9BAE38F000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F9BAE179000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F9BADF5B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F9BADB96000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F9BAF0DC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f9bb0880221]
       mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x7f9bb087f734]
       mongod(+0x173C2CB) [0x7f9bb0a3d2cb]
       mongod(tc_newarray_nothrow+0x222) [0x7f9bb1377072]
       mongod(_ZSt20get_temporary_bufferISt4pairIN5mongo7BSONObjENS1_8RecordIdEEES0_IPT_lEl+0x45) [0x7f9baff76cb5]
       mongod(_ZNSt17_Temporary_bufferISt15_Deque_iteratorISt4pairIN5mongo7BSONObjENS2_8RecordIdEERS5_PS5_ES5_EC1ES8_S8_+0x72) [0x7f9baff76d52]
       mongod(_ZN5mongo6sorter13NoLimitSorterINS_7BSONObjENS_8RecordIdENS_27BtreeExternalSortComparisonEE4sortEv+0xBA) [0x7f9baff7df8a]
       mongod(_ZN5mongo6sorter13NoLimitSorterINS_7BSONObjENS_8RecordIdENS_27BtreeExternalSortComparisonEE4doneEv+0x4D) [0x7f9baff7e61d]
       mongod(_ZN5mongo17IndexAccessMethod10commitBulkEPNS_16OperationContextESt10unique_ptrINS0_11BulkBuilderESt14default_deleteIS4_EEbbPSt3setINS_8RecordIdESt4lessIS9_ESaIS9_EE+0x79) [0x7f9baff6d749]
       mongod(_ZN5mongo15MultiIndexBlock13doneInsertingEPSt3setINS_8RecordIdESt4lessIS2_ESaIS2_EE+0x1DB) [0x7f9bafd2076b]
       mongod(+0xF17A91) [0x7f9bb0218a91]
       mongod(+0xF18673) [0x7f9bb0219673]
       mongod(+0xF186F5) [0x7f9bb02196f5]
       mongod(+0x1068341) [0x7f9bb0369341]
       mongod(+0x1068577) [0x7f9bb0369577]
       mongod(_ZN5mongo4repl10TaskRunner9_runTasksEv+0xAF) [0x7f9bb036a75f]
       mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x135) [0x7f9bb07f72c5]
       mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xC0) [0x7f9bb07f7df0]
       mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x149) [0x7f9bb07f8999]
       mongod(+0x1FF41D0) [0x7f9bb12f51d0]
       libpthread.so.0(+0x8184) [0x7f9badf63184]
       libc.so.6(clone+0x6D) [0x7f9badc9037d]
      -----  END BACKTRACE  -----
      

      Any help will be appreciated.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              kelsey.schubert Kelsey T Schubert
              Reporter:
              thevivekpandey Vivek Pandey
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: