Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-13794

Unused snapshot history consuming significant heap space

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Done
    • 2.6.0
    • 2.6.2, 2.7.1
    • Diagnostics
    • None
    • Fully Compatible
    • ALL
    • Server 2.7.1

    Description

      Issue Status as of May 16, 2014

      ISSUE SUMMARY
      mongod consumes significant heap space on unused snapshot history when snapshots are enabled. This increased memory use is not a leak, and is proportional to the number of collections.

      USER IMPACT
      Users will see increased memory usage by mongod processes.

      WORKAROUNDS
      N/A

      AFFECTED VERSIONS
      All 2.4 and 2.6 MongoDB production releases are affected by this isse.

      FIX VERSION
      The fix is included in the 2.6.2 production release.

      RESOLUTION DETAILS
      The snapshot thread, which provides historical collection level and lock statistics for use by the web interface, is only needed when HTTP is enabled. To avoid this extra memory consumption, disable the HTTP interface.

      Original description

      We saw a case where an upgrade from 2.4 to 2.6 led to non-mapped memory peak growing for all 2.6.0 members of a replica set. Taking the data set in question and 15 minute sampling of read operations I was able to reproduce 200MB growth. Heap profiling in Valgrind led to discovery that this memory was being consumed by the snapshots thread (db/stats/snapshots.h/.cpp).

      The snapshots thread currently maintains an array of 100 snapshots. Each snapshot contains collection level usage data, the size of which grows with each collection contained within a mongod instance. The data set used to reproduce contains 3588 collections.

      The growth occurs in the following valgrind massif stack. The full ms_print valgrind output will be attached as well.

      99.41% (153,881,596B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
      ->55.57% (86,016,800B) 0x138BE28: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::Area::Area(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::Area const&) (unordered_fast_key_table_internal.h:34)
      | ->55.57% (86,016,800B) 0x138BC40: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::copyTo(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>*) const (unordered_fast_key_table_internal.h:127)
      |   ->55.57% (86,016,800B) 0x138BB85: mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther>::operator=(mongo::UnorderedFastKeyTable<mongo::StringData, std::string, mongo::Top::CollectionData, mongo::StringData::Hasher, mongo::StringMapDefaultEqual, mongo::StringMapDefaultConvertor, mongo::StringMapDefaultConvertorOther> const&) (unordered_fast_key_table.h:94)
      |     ->55.57% (86,016,800B) 0x138B7B1: mongo::StringMap<mongo::Top::CollectionData>::operator=(mongo::StringMap<mongo::Top::CollectionData> const&) (string_map.h:46)
      |       ->55.57% (86,016,800B) 0x138AFCC: mongo::Top::cloneMap(mongo::StringMap<mongo::Top::CollectionData>&) const (top.cpp:130)
      |         ->55.57% (86,016,800B) 0x13878C4: mongo::SnapshotData::takeSnapshot() (snapshots.cpp:47)
      |           ->55.57% (86,016,800B) 0x1387E03: mongo::Snapshots::takeSnapshot() (snapshots.cpp:83)
      |             ->55.57% (86,016,800B) 0x13881D6: mongo::SnapshotThread::run() (snapshots.cpp:128)
      |               ->55.57% (86,016,800B) 0x151E2B8: mongo::BackgroundJob::jobBody() (background.cpp:147)
      |                 ->55.57% (86,016,800B) 0x1521793: boost::_mfi::mf0<void, mongo::BackgroundJob>::operator()(mongo::BackgroundJob*) const (mem_fn_template.hpp:49)
      |                   ->55.57% (86,016,800B) 0x15216F6: void boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> >::operator()<boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, mongo::BackgroundJob>&, boost::_bi::list0&, int) (bind.hpp:253)
      |                     ->55.57% (86,016,800B) 0x15216A5: boost::_bi::bind_t<void, boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> > >::operator()() (bind_template.hpp:20)
      |                       ->55.57% (86,016,800B) 0x152166A: boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void, mongo::BackgroundJob>, boost::_bi::list1<boost::_bi::value<mongo::BackgroundJob*> > > >::run() (thread.hpp:62)
      |                         ->55.57% (86,016,800B) 0x15C11E8: thread_proxy (thread.cpp:121)
      |                           ->55.57% (86,016,800B) 0x4E36180: start_thread (pthread_create.c:312)
      |                             ->55.57% (86,016,800B) 0x596730B: clone (clone.S:111)
      |                               

      Attachments

        Issue Links

          Activity

            People

              james.wahlin@mongodb.com James Wahlin
              james.wahlin@mongodb.com James Wahlin
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: