Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26753

Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2

    Details

    • Type: Task
    • Status: Closed
    • Priority: Minor - P4
    • Resolution: Fixed
    • Affects Version/s: 3.4.0-rc1
    • Fix Version/s: 3.2.11, 3.4.0-rc3
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Backport Completed:
    • Sprint:
      Storage 2016-11-21

      Description

      OS is Win2012R2.

      When comparing my workload on the 3.4rc1 vs 3.2.10 there is a minor speed regression (13%) which appears to be mainly due to some intermittent activity (eviction?).

      I see the cpu spike to 100% every ~10-20 seconds with a corresponding speed drop. In 3.2, the cpu stays relatively flat.

      This is minor but thought you might like to know.

      1. 3.2.10.metrics.2016-10-26T15-56-24Z-00000
        197 kB
        Nick Judson
      2. 3.2.9-high-cpu.metrics.2016-11-04T16-46-06Z-00000
        114 kB
        Nick Judson
      3. 3.2.metrics.2016-10-24T19-17-12Z-00000
        119 kB
        Nick Judson
      4. 3.4.metrics.2016-10-24T19-06-15Z-00000
        164 kB
        Nick Judson
      5. high-cpu.metrics.2016-11-04T16-20-56Z-00000
        65 kB
        Nick Judson
      6. high-cpu2.metrics.2016-11-04T16-37-01Z-00000
        109 kB
        Nick Judson
      1. Custom Build.png
        350 kB
      2. mongo.3.2.10.png
        283 kB
      3. mongo.3.2.png
        360 kB
      4. mongo3.4.png
        383 kB

        Activity

        Hide
        nick@innsenroute.com Nick Judson added a comment -

        Must be something to do with my environment since I see a similar behavior in 3.2.9. Interesting to note that 3.2.9 still shows a better performance / lower cpu usage. I'm attaching the 3.2.9 run.

        Show
        nick@innsenroute.com Nick Judson added a comment - Must be something to do with my environment since I see a similar behavior in 3.2.9. Interesting to note that 3.2.9 still shows a better performance / lower cpu usage. I'm attaching the 3.2.9 run.
        Hide
        nick@innsenroute.com Nick Judson added a comment -

        ...and after rebooting the MongoDb host, it appears to have reverted back to the behavior I'm used to...

        Show
        nick@innsenroute.com Nick Judson added a comment - ...and after rebooting the MongoDb host, it appears to have reverted back to the behavior I'm used to...
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

        Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4

        ref: ef9a7983ea..74430da40c
        for: 3.4.0

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw compression to the WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: master
        https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4 ref: ef9a7983ea..74430da40c for: 3.4.0 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw compression to the WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: master https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
        Hide
        michael.cahill Michael Cahill added a comment -

        Nick Judson, thanks for the update: this is a little puzzling.

        We can see clearly in the stats that the read lock spinning has been resolved: in 3.2.9 your data shows occasional spikes in this statistic of 150M/s, in 3.2.10 there were many more spikes going up to 450M/s. With the patch applied, this stat never goes above 166K/s (i.e., 1000x lower).

        I'm not sure what to make of the lower performance: we don't track system statistics such as CPU and disk in MongoDB 3.2, so I can't correlate performance with CPU use from this data.

        I am going to close this ticket because I believe the underlying issue has been resolved by changes that will be in the next release of MongoDB. If you want more followup, please reopen or file a new ticket.

        Show
        michael.cahill Michael Cahill added a comment - Nick Judson , thanks for the update: this is a little puzzling. We can see clearly in the stats that the read lock spinning has been resolved: in 3.2.9 your data shows occasional spikes in this statistic of 150M/s, in 3.2.10 there were many more spikes going up to 450M/s. With the patch applied, this stat never goes above 166K/s (i.e., 1000x lower). I'm not sure what to make of the lower performance: we don't track system statistics such as CPU and disk in MongoDB 3.2, so I can't correlate performance with CPU use from this data. I am going to close this ticket because I believe the underlying issue has been resolved by changes that will be in the next release of MongoDB. If you want more followup, please reopen or file a new ticket.
        Hide
        xgen-internal-githook Githook User added a comment -

        Author:

        {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

        Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2

        ref: b11ed312ce..2e2d5fe23b
        for: 3.2.11

        SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
        WT-2415 Add option for join cursors to return false positives from a Bloom filter
        WT-2945 Occasional hang running reconfigure fuzz test
        WT-2947 test suite populate functions need refactoring
        WT-2953 Test format failure - stuck cache with 1/42MB dirty
        WT-2961 Add a version drop-down to the web version of the docs
        WT-2964 WTPERF parallel-pop-btree hung in populate phase
        WT-2968 backup cursor opens every file
        WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
        WT-2975 Reduce the memory allocated for statistics
        WT-2991 Coverity issue 1365198
        WT-2998 Add error messages to error returns that might be confusing.
        WT-2999 Memory leak in join WT_CURSOR with custom extractor
        WT-3001 WT_EXTENSION_API references are named inconsistently.
        WT-3003 log generation should not auto generate the record and op types
        WT-3005 Add top-level .gitignore file.
        WT-3007 Remove duplicated word in comment
        WT-3008 Run wtperf stress testing against all maintained branches
        WT-3009 Test format hung/aborted with small cache (sub 10mB)
        Branch: v3.2
        https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

        Show
        xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2 ref: b11ed312ce..2e2d5fe23b for: 3.2.11 SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2 WT-2415 Add option for join cursors to return false positives from a Bloom filter WT-2945 Occasional hang running reconfigure fuzz test WT-2947 test suite populate functions need refactoring WT-2953 Test format failure - stuck cache with 1/42MB dirty WT-2961 Add a version drop-down to the web version of the docs WT-2964 WTPERF parallel-pop-btree hung in populate phase WT-2968 backup cursor opens every file WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs WT-2975 Reduce the memory allocated for statistics WT-2991 Coverity issue 1365198 WT-2998 Add error messages to error returns that might be confusing. WT-2999 Memory leak in join WT_CURSOR with custom extractor WT-3001 WT_EXTENSION_API references are named inconsistently. WT-3003 log generation should not auto generate the record and op types WT-3005 Add top-level .gitignore file. WT-3007 Remove duplicated word in comment WT-3008 Run wtperf stress testing against all maintained branches WT-3009 Test format hung/aborted with small cache (sub 10mB) Branch: v3.2 https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

                Agile