Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2868

Add sample_interval to checkpoint-stress wtperf config

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.3.12, 3.2.10
    • Labels:
      None
    • # Replies:
      12
    • Last comment by Customer:
      true

      Description

      Add sample_interval to generate monitor file for gathering throughput and latency stats

        Activity

        Hide
        sue.loverso Sue LoVerso added a comment -

        FYI I hand-edited the csv files so that the plots are consistently labeled with your new labels and they look like you'd expect.

        Show
        sue.loverso Sue LoVerso added a comment - FYI I hand-edited the csv files so that the plots are consistently labeled with your new labels and they look like you'd expect.
        Hide
        sulabh.mahajan Sulabh Mahajan added a comment -

        Sue Loverso thanks for the input, I will work on this and get back to you.

        Show
        sulabh.mahajan Sulabh Mahajan added a comment - Sue Loverso thanks for the input, I will work on this and get back to you.
        Hide
        sulabh.mahajan Sulabh Mahajan added a comment -

        Sue Loverso, the changes to the wtperf file are under review::

        • We decided to keep the sample* lines for any use in the future. I can remove them if you feel otherwise.
        • We are not necessarily imitating how MongoDB uses WT, I have added eviction threads and increased cache size.
        • Since this test is partly intended for the performance measurements, I am inclined to not turn on statistics till needed. Let me know if you feel otherwise
        • This wtperf configuration is mostly based on the test that was used for WT-2389, to keep track of any performance regression in no of updates with stressed checkpoints.
          I am not sure if this goes well with any other tests. For now I am inclined towards not merging this with any other test.
        Show
        sulabh.mahajan Sulabh Mahajan added a comment - Sue Loverso , the changes to the wtperf file are under review: : We decided to keep the sample* lines for any use in the future. I can remove them if you feel otherwise. We are not necessarily imitating how MongoDB uses WT, I have added eviction threads and increased cache size. Since this test is partly intended for the performance measurements, I am inclined to not turn on statistics till needed. Let me know if you feel otherwise This wtperf configuration is mostly based on the test that was used for WT-2389 , to keep track of any performance regression in no of updates with stressed checkpoints. I am not sure if this goes well with any other tests. For now I am inclined towards not merging this with any other test.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'sulabhM', u'name': u'Sulabh Mahajan', u'email': u'sulabh.mahajan@mongodb.com'}

        Message: WT-2868 Add sample_interval to checkpoint-stress.wtperf (#2989)
        Branch: mongodb-3.2
        https://github.com/wiredtiger/wiredtiger/commit/f07c61492f69a45d566980a31b46f0044093bb37

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'sulabhM', u'name': u'Sulabh Mahajan', u'email': u'sulabh.mahajan@mongodb.com'} Message: WT-2868 Add sample_interval to checkpoint-stress.wtperf (#2989) Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/f07c61492f69a45d566980a31b46f0044093bb37
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

        Message: Import wiredtiger: 911c940adab547d36ac305fc627a79e637fa3c40 from branch mongodb-3.2

        ref: dddca65..911c940ada
        for: 3.2.10

        SERVER-24971 Excessive memory held by sessions when application threads do evictions
        SERVER-25843 Coverity analysis defect 99856: Redundant test
        SERVER-25845 Coverity analysis defect 99859: Explicit null dereferenced
        SERVER-25846 Coverity analysis defect 99861: Dereference after null check
        WT-1162 Add latency to Jenkins wtperf tests and plots
        WT-2026 Maximum pages size at eviction too large
        WT-2221 Document which statistics are available via a "fast" configuration vs. an "all" configuration
        WT-2233 Investigate changing when the eviction server switches to aggressive mode.
        WT-2239 Make sure LSM cursors read up to date dsk_gen, it was racing with compact
        WT-2323 Allocate a transaction id at the beginning of join cursor iteration
        WT-2353 Failure to create async threads as part of a wiredtiger_open call will cause a hang
        WT-2380 Make scripts fail if code doesn't match style
        WT-2486 Update make check so that it runs faster
        WT-2555 make format run on Windows
        WT-2578 remove write barriers from the TAILQ_INSERT_XXX macros
        WT-2631 nullptr is passed for parameters marked with attribute non-null
        WT-2638 ftruncate may not be supported
        WT-2645 wt dump: push the complexity of collecting metadata into a dump cursor
        WT-2648 cache-line alignment for new ports
        WT-2665 Limit allocator fragmentation in WiredTiger
        WT-2678 The metadata should not imply that an empty value is true
        WT-2688 configure --enable-python doesn't check for availability of swig
        WT-2693 Check open_cursor error paths for consistent handling
        WT-2695 Integrate s390x accelerated crc32c support
        WT-2708 split child-update race with reconciliation/eviction
        WT-2711 Change statistics log configuration options
        WT-2719 add fuzz testing for WiredTiger options and reconfiguration.
        WT-2728 Don't re-read log file headers during log_flush
        WT-2729 Focus eviction walks in largest trees
        WT-2730 cursor next/prev can return the wrong key/value pair when crossing a page boundary
        WT-2731 Raw compression can create pages that are larger than expected
        WT-2732 Coverity analysis defect 99665: Redundant test
        WT-2734 Improve documentation of eviction behavior
        WT-2737 Scrub dirty pages rather than evicting them
        WT-2738 Remove the ability to change the default checkpoint name
        WT-2739 pluggable file systems documentation cleanups
        WT-2743 Thread count statistics always report 0
        WT-2744 partial line even with line buffering set
        WT-2746 track checkpoint I/O separately from eviction I/O
        WT-2751 column-store statistics incorrectly calculates the number of entries
        WT-2752 Fixes to zipfian wtperf workload config
        WT-2755 flexelint configuration treats size_t as 4B type
        WT-2756 Upgrade the autoconf archive package to check for swig 3.0
        WT-2757 Column tables behave differently when column names are provided
        WT-2759 Releasing the hot-backup lock doesn't require the schema lock.
        WT-2760 Fix a bug in backup related to directory sync. Change the filesystem API to make durable the default
        WT-2762 wtstats tool fails if checkpoint runs
        WT-2763 Unit test test_intpack failing on OSX
        WT-2764 Optimize checkpoints to reduce throughput disruption
        WT-2765 wt dump: indices need to be shown in the dump output
        WT-2766 Don't count eviction of lookaside file pages for the purpose of checking stuck cache
        WT-2767 test suite needs way to run an individual scenario
        WT-2769 Update documentation to reflect correct limits of memory_page_max
        WT-2770 Add statistics tracking schema operations
        WT-2772 Investigate log performance testing weirdness
        WT-2773 search_near in indexes does not find exact matches
        WT-2774 minor cleanups/improvements
        WT-2778 Python test suite: make scenario initialization consistent
        WT-2779 Raw compression created unexpectedly large pages on disk
        WT-2781 Enhance bulk cursor option with an option to return immediately on contention
        WT-2782 Missing a fs_directory_list_free in ex_file_system.c
        WT-2783 wtperf multi-btree.wtperf dumps core on Mac
        WT-2785 Scrub dirty pages rather than evicting them: single-page reconciliation
        WT-2787 Include src/include/wiredtiger_ext.h is problematic
        WT-2788 Java: freed memory overwrite during handle close can cause JNI crash
        WT-2791 Enhance OS X Evergreen unit test
        WT-2793 wtperf config improvements
        WT-2795 Update documentation around read-only configuration
        WT-2796 Memory leak in reconciliation uncovered by stress testing
        WT-2798 Crash vulnerability with nojournal after create during checkpoint
        WT-2800 Illegal file format in test/format on PPC
        WT-2801 Crash vulnerability from eviction of metadata during checkpoint
        WT-2802 Transaction commit causes heap-use-after free
        WT-2803 Add verbose functionality to WT Evergreen tests
        WT-2804 Don't read values in a tree without a snapshot
        WT-2805 Infinite recursion if error streams fail
        WT-2806 wtperf allocation size off-by-one
        WT-2807 Switch Jenkins performance tests to tcmalloc
        WT-2811 Reconciliation asserts that transaction time has gone backwards
        WT-2812 Error when reconfiguring cache targets
        WT-2813 small cache usage stuck even with large cache
        WT-2814 Enhance wtperf to support single-op truncate mode
        WT-2816 Improve WiredTiger eviction performance
        WT-2817 Investigate performance regression in develop, add workload to wtperf/runners
        WT-2818 The page visibility check when queuing pages for eviction is overly restrictive
        WT-2820 add gcc warn_unused_result attribute
        WT-2822 panic mutex and other functions that cannot fail
        WT-2823 support file handles without a truncate method
        WT-2824 wtperf displays connection and table create configurations twice
        WT-2826 clang38 false positive on uninitialized variable.
        WT-2827 checkpoint log_size configuration improvements
        WT-2828 Make long wtperf tests reflect mongoDB usage
        WT-2829 Switch automated testing to use enable-strict configure option
        WT-2832 Python test uses hard-coded temporary directory
        WT-2834 Join cursor: discrepancy with bloom filters
        WT-2835 WT_CONNECTION.leak-memory can skip memory map and cache cleanup
        WT-2838 Don't free session handles on close if leak memory is configured
        WT-2839 lint: Ignoring return value of function
        WT-2840 clang analysis: garbage values
        WT-2841 Jenkins Valgrind runner is reporting errors in test wt2719_reconfig
        WT-2842 split wtperf's configuration into per-database and per-run parts
        WT-2843 Fix a bug in recovery if there is no filesystem truncate support
        WT-2846 Several bugs related to reconfiguring eviction server at runtime
        WT-2847 Merge fair locks into read/write locks.
        WT-2850 clang 4.1 attribute warnings when building
        WT-2853 Multi threaded reader writer example shows temporary slowdown or lockup
        WT-2857 POSIX ftruncate calls should be #ifdef'd HAVE_FTRUNCATE
        WT-2862 Fix lint error in test case for forced eviction with multiple cursors
        WT-2863 Support UTF-8 paths on Windows
        WT-2865 eviction thread error failure
        WT-2866 Eviction server algorithm tuning
        WT-2867 Review and fix barrier usage in __lsm_tree_close
        WT-2868 Add sample_interval to checkpoint-stress wtperf config
        WT-2869 Performance regression on secondaries
        WT-2870 Rename wtperf checkpoint schema jobs
        WT-2871 __wt_verbose has the wrong GCC format attributes
        WT-2872 Recent stuck cache test/stress failures.
        WT-2873 Refactor CRC32 code
        WT-2875 Test test_wt2853_perf can run too long under valgrind
        WT-2876 Extend wtperf to support a log like table
        WT-2878 Verbose changes affected performance
        WT-2881 Add -Wpedantic to clang compiler warning flags
        WT-2883 wiredtiger_open with verbose=handleops recursive loop
        WT-2885 __wt_checkpoint_signal lint
        WT-2886 Decide how in-memory configuration and eviction_dirty_target interact
        WT-2888 Switch functions to return void where possible
        WT-2892 hot backup can race with block truncate
        WT-2896 Coverity #1362535: resource leak
        WT-2897 Checkpoints can become corrupted on failure
        WT-2901 Add option to disable checkpoint dirty stepdown phase
        WT-2903 Reduce the impact of checkpoint scrubbing on applications
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/7d2acd6395ec84beca34718a75371bc11f0c9f60

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: 911c940adab547d36ac305fc627a79e637fa3c40 from branch mongodb-3.2 ref: dddca65..911c940ada for: 3.2.10 SERVER-24971 Excessive memory held by sessions when application threads do evictions SERVER-25843 Coverity analysis defect 99856: Redundant test SERVER-25845 Coverity analysis defect 99859: Explicit null dereferenced SERVER-25846 Coverity analysis defect 99861: Dereference after null check WT-1162 Add latency to Jenkins wtperf tests and plots WT-2026 Maximum pages size at eviction too large WT-2221 Document which statistics are available via a "fast" configuration vs. an "all" configuration WT-2233 Investigate changing when the eviction server switches to aggressive mode. WT-2239 Make sure LSM cursors read up to date dsk_gen, it was racing with compact WT-2323 Allocate a transaction id at the beginning of join cursor iteration WT-2353 Failure to create async threads as part of a wiredtiger_open call will cause a hang WT-2380 Make scripts fail if code doesn't match style WT-2486 Update make check so that it runs faster WT-2555 make format run on Windows WT-2578 remove write barriers from the TAILQ_INSERT_XXX macros WT-2631 nullptr is passed for parameters marked with attribute non-null WT-2638 ftruncate may not be supported WT-2645 wt dump: push the complexity of collecting metadata into a dump cursor WT-2648 cache-line alignment for new ports WT-2665 Limit allocator fragmentation in WiredTiger WT-2678 The metadata should not imply that an empty value is true WT-2688 configure --enable-python doesn't check for availability of swig WT-2693 Check open_cursor error paths for consistent handling WT-2695 Integrate s390x accelerated crc32c support WT-2708 split child-update race with reconciliation/eviction WT-2711 Change statistics log configuration options WT-2719 add fuzz testing for WiredTiger options and reconfiguration. WT-2728 Don't re-read log file headers during log_flush WT-2729 Focus eviction walks in largest trees WT-2730 cursor next/prev can return the wrong key/value pair when crossing a page boundary WT-2731 Raw compression can create pages that are larger than expected WT-2732 Coverity analysis defect 99665: Redundant test WT-2734 Improve documentation of eviction behavior WT-2737 Scrub dirty pages rather than evicting them WT-2738 Remove the ability to change the default checkpoint name WT-2739 pluggable file systems documentation cleanups WT-2743 Thread count statistics always report 0 WT-2744 partial line even with line buffering set WT-2746 track checkpoint I/O separately from eviction I/O WT-2751 column-store statistics incorrectly calculates the number of entries WT-2752 Fixes to zipfian wtperf workload config WT-2755 flexelint configuration treats size_t as 4B type WT-2756 Upgrade the autoconf archive package to check for swig 3.0 WT-2757 Column tables behave differently when column names are provided WT-2759 Releasing the hot-backup lock doesn't require the schema lock. WT-2760 Fix a bug in backup related to directory sync. Change the filesystem API to make durable the default WT-2762 wtstats tool fails if checkpoint runs WT-2763 Unit test test_intpack failing on OSX WT-2764 Optimize checkpoints to reduce throughput disruption WT-2765 wt dump: indices need to be shown in the dump output WT-2766 Don't count eviction of lookaside file pages for the purpose of checking stuck cache WT-2767 test suite needs way to run an individual scenario WT-2769 Update documentation to reflect correct limits of memory_page_max WT-2770 Add statistics tracking schema operations WT-2772 Investigate log performance testing weirdness WT-2773 search_near in indexes does not find exact matches WT-2774 minor cleanups/improvements WT-2778 Python test suite: make scenario initialization consistent WT-2779 Raw compression created unexpectedly large pages on disk WT-2781 Enhance bulk cursor option with an option to return immediately on contention WT-2782 Missing a fs_directory_list_free in ex_file_system.c WT-2783 wtperf multi-btree.wtperf dumps core on Mac WT-2785 Scrub dirty pages rather than evicting them: single-page reconciliation WT-2787 Include src/include/wiredtiger_ext.h is problematic WT-2788 Java: freed memory overwrite during handle close can cause JNI crash WT-2791 Enhance OS X Evergreen unit test WT-2793 wtperf config improvements WT-2795 Update documentation around read-only configuration WT-2796 Memory leak in reconciliation uncovered by stress testing WT-2798 Crash vulnerability with nojournal after create during checkpoint WT-2800 Illegal file format in test/format on PPC WT-2801 Crash vulnerability from eviction of metadata during checkpoint WT-2802 Transaction commit causes heap-use-after free WT-2803 Add verbose functionality to WT Evergreen tests WT-2804 Don't read values in a tree without a snapshot WT-2805 Infinite recursion if error streams fail WT-2806 wtperf allocation size off-by-one WT-2807 Switch Jenkins performance tests to tcmalloc WT-2811 Reconciliation asserts that transaction time has gone backwards WT-2812 Error when reconfiguring cache targets WT-2813 small cache usage stuck even with large cache WT-2814 Enhance wtperf to support single-op truncate mode WT-2816 Improve WiredTiger eviction performance WT-2817 Investigate performance regression in develop, add workload to wtperf/runners WT-2818 The page visibility check when queuing pages for eviction is overly restrictive WT-2820 add gcc warn_unused_result attribute WT-2822 panic mutex and other functions that cannot fail WT-2823 support file handles without a truncate method WT-2824 wtperf displays connection and table create configurations twice WT-2826 clang38 false positive on uninitialized variable. WT-2827 checkpoint log_size configuration improvements WT-2828 Make long wtperf tests reflect mongoDB usage WT-2829 Switch automated testing to use enable-strict configure option WT-2832 Python test uses hard-coded temporary directory WT-2834 Join cursor: discrepancy with bloom filters WT-2835 WT_CONNECTION.leak-memory can skip memory map and cache cleanup WT-2838 Don't free session handles on close if leak memory is configured WT-2839 lint: Ignoring return value of function WT-2840 clang analysis: garbage values WT-2841 Jenkins Valgrind runner is reporting errors in test wt2719_reconfig WT-2842 split wtperf's configuration into per-database and per-run parts WT-2843 Fix a bug in recovery if there is no filesystem truncate support WT-2846 Several bugs related to reconfiguring eviction server at runtime WT-2847 Merge fair locks into read/write locks. WT-2850 clang 4.1 attribute warnings when building WT-2853 Multi threaded reader writer example shows temporary slowdown or lockup WT-2857 POSIX ftruncate calls should be #ifdef'd HAVE_FTRUNCATE WT-2862 Fix lint error in test case for forced eviction with multiple cursors WT-2863 Support UTF-8 paths on Windows WT-2865 eviction thread error failure WT-2866 Eviction server algorithm tuning WT-2867 Review and fix barrier usage in __lsm_tree_close WT-2868 Add sample_interval to checkpoint-stress wtperf config WT-2869 Performance regression on secondaries WT-2870 Rename wtperf checkpoint schema jobs WT-2871 __wt_verbose has the wrong GCC format attributes WT-2872 Recent stuck cache test/stress failures. WT-2873 Refactor CRC32 code WT-2875 Test test_wt2853_perf can run too long under valgrind WT-2876 Extend wtperf to support a log like table WT-2878 Verbose changes affected performance WT-2881 Add -Wpedantic to clang compiler warning flags WT-2883 wiredtiger_open with verbose=handleops recursive loop WT-2885 __wt_checkpoint_signal lint WT-2886 Decide how in-memory configuration and eviction_dirty_target interact WT-2888 Switch functions to return void where possible WT-2892 hot backup can race with block truncate WT-2896 Coverity #1362535: resource leak WT-2897 Checkpoints can become corrupted on failure WT-2901 Add option to disable checkpoint dirty stepdown phase WT-2903 Reduce the impact of checkpoint scrubbing on applications Branch: v3.2 https://github.com/mongodb/mongo/commit/7d2acd6395ec84beca34718a75371bc11f0c9f60

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              27 weeks, 2 days ago
              Date of 1st Reply: