Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20193

Performance regression in Single-threaded MMS workloads from git: a7338c4

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.8
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      Big performance drop (>80%) from git: a7338c4 on single-threaded MMS workloads. Same build was marked red for Geo and Misc cases but there only one test failed for single-threaded cases and the drop was much smaller. I think it's safe to ignore that and just focus on the MMS cases.

      The commit that caused this was a wiredTiger check-in. WT team is aware of this drop so I am putting WT as the component for this ticket so they can pick this ticket up.

        Issue Links

          Activity

          Hide
          michael.cahill Michael Cahill added a comment -

          https://evergreen.mongodb.com/task/performance_linux_wt_repl_singleThreaded_a7338c436160e259d0b20058f0a9ae4b442ca102_15_08_28_06_06_32

          This is the same issue as SERVER-19445 – if the oldest transaction ID tracked by WiredTiger falls behind, workloads that do many updates to medium-large documents to hit forced eviction. The regression was introduced by WT-2066.

          Show
          michael.cahill Michael Cahill added a comment - https://evergreen.mongodb.com/task/performance_linux_wt_repl_singleThreaded_a7338c436160e259d0b20058f0a9ae4b442ca102_15_08_28_06_06_32 This is the same issue as SERVER-19445 – if the oldest transaction ID tracked by WiredTiger falls behind, workloads that do many updates to medium-large documents to hit forced eviction. The regression was introduced by WT-2066 .
          Hide
          michael.cahill Michael Cahill added a comment -

          David Hows, I'm pretty sure my comment above explains what's going on here – can you please re-run this and make sure you can see the slowdown?

          The fix is to call __wt_txn_update_oldest more frequently in this workload, for some definition of "more frequently", so that obsolete updates can be freed before the page size hits memory_page_max.

          For example, we change the end of __wt_update_serial in btree.i to something like this:

          if (upd->next == NULL)
              return (0);
           
          if (page->modify->obsolete_check_txn != WT_TXN_NONE) {
              if (!__wt_txn_visible_all(session, page->modify->obsolete_check_txn)) {
                  /* Check if the oldest ID can move forward and re-check. */
                  __wt_txn_update_oldest(session);
                  if (!__wt_txn_visible_all(session, page->modify->obsolete_check_txn))
                      return (0);
          }
           
          F_CAS_ATOMIC(page, WT_PAGE_RECONCILIATION, ret);
          /* If we can't lock it, don't scan, that's okay. */
          if (ret != 0)
              return (0);
          obsolete = __wt_update_obsolete_check(session, page, upd->next);
          ...
          

          Does that make sense?

          Show
          michael.cahill Michael Cahill added a comment - David Hows , I'm pretty sure my comment above explains what's going on here – can you please re-run this and make sure you can see the slowdown? The fix is to call __wt_txn_update_oldest more frequently in this workload, for some definition of "more frequently", so that obsolete updates can be freed before the page size hits memory_page_max . For example, we change the end of __wt_update_serial in btree.i to something like this: if (upd->next == NULL) return (0);   if (page->modify->obsolete_check_txn != WT_TXN_NONE) { if (!__wt_txn_visible_all(session, page->modify->obsolete_check_txn)) { /* Check if the oldest ID can move forward and re-check. */ __wt_txn_update_oldest(session); if (!__wt_txn_visible_all(session, page->modify->obsolete_check_txn)) return (0); }   F_CAS_ATOMIC(page, WT_PAGE_RECONCILIATION, ret); /* If we can't lock it, don't scan, that's okay. */ if (ret != 0) return (0); obsolete = __wt_update_obsolete_check(session, page, upd->next); ... Does that make sense?
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

          Message: SERVER-20193 - Performance regression fix
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/713daf2a75a6c602e9ffa23f74dfec2f0ddc9a8a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: SERVER-20193 - Performance regression fix Branch: develop https://github.com/wiredtiger/wiredtiger/commit/713daf2a75a6c602e9ffa23f74dfec2f0ddc9a8a
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

          Message: SERVER-20193 - Change when WT_TXN_NONE is set, update comments
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/a60a16cbf14f67c4b6a4c4452782f5faac1617cc

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'} Message: SERVER-20193 - Change when WT_TXN_NONE is set, update comments Branch: develop https://github.com/wiredtiger/wiredtiger/commit/a60a16cbf14f67c4b6a4c4452782f5faac1617cc
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2177 from wiredtiger/SERVER-20193

          SERVER-20193 - Performance regression fix
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/b044603cf9c3032526ddc144fdf23ef9d862b617

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2177 from wiredtiger/ SERVER-20193 SERVER-20193 - Performance regression fix Branch: develop https://github.com/wiredtiger/wiredtiger/commit/b044603cf9c3032526ddc144fdf23ef9d862b617
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2177 from wiredtiger/SERVER-20193

          SERVER-20193 - Performance regression fix
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/b044603cf9c3032526ddc144fdf23ef9d862b617

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2177 from wiredtiger/ SERVER-20193 SERVER-20193 - Performance regression fix Branch: develop https://github.com/wiredtiger/wiredtiger/commit/b044603cf9c3032526ddc144fdf23ef9d862b617
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-20193 Fix clearing of obsolete check transaction.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/0b359e0a3661f859db8d5d86c255045582f5d556

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-20193 Fix clearing of obsolete check transaction. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/0b359e0a3661f859db8d5d86c255045582f5d556
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2188 from wiredtiger/visible_all_call

          SERVER-20193 Fix obsolete transaction check
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/8be547b318f0195d0c3c13a692589631f381f935

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2188 from wiredtiger/visible_all_call SERVER-20193 Fix obsolete transaction check Branch: develop https://github.com/wiredtiger/wiredtiger/commit/8be547b318f0195d0c3c13a692589631f381f935

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: