Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1953

__compact_file can perform hundreds of unnecessary checkpoints

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.6.1
    • Labels:

      Description

      __compact_file has a loop that can perform hundreds of checkpoints during a single compact command:

      /*
       * We compact 10% of the file on each pass (but the overall size of the
       * file is decreasing each time, so we're not compacting 10% of the
       * original file each time). Try 100 times (which is clearly more than
       * we need); quit if we make no progress and check for a timeout each
       * time through the loop.
       */
      for (i = 0; i < 100; ++i) {
          WT_ERR(wt_session->checkpoint(wt_session, t->data));
       
          session->compaction = 0;
          WT_WITH_SCHEMA_LOCK(session,
              ret = __wt_schema_worker(
              session, uri, __wt_compact, NULL, cfg, 0));
          WT_ERR(ret);
          if (!session->compaction)
              break;
       
          WT_ERR(wt_session->checkpoint(wt_session, t->data));
          WT_ERR(wt_session->checkpoint(wt_session, t->data));
          WT_ERR(__session_compact_check_timeout(session, start_time));
      }
      

      This worst case scenario is hit 100 times in the JS test below:

      'use strict';
       
      var t = db.compact;
      t.drop();
       
      t.insert({});
       
      for (var i = 0; i < 100; i++) {
          var res = db.runCommand({compact: t.getName()});
          assert.commandWorked(res);
      }
      


      Version info:
      db version v3.1.4-pre-
      git version: 845a2dcc23a597520065d1258089cc87dabf6cb8

        Issue Links

          Activity

          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          The issue here is with the tests WiredTiger has internally to determine whether it is worthwhile to continue compacting, and whether further iterations of compact are making progress.

          A compact on the file in this test case should be a no-op since it only has a single record in it.

          Show
          alexander.gorrod Alexander Gorrod added a comment - The issue here is with the tests WiredTiger has internally to determine whether it is worthwhile to continue compacting, and whether further iterations of compact are making progress. A compact on the file in this test case should be a no-op since it only has a single record in it.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: Only set the compaction flag once we find a page to compact.

          Otherwise files with a single page can end up having multiple
          checkpoints run on them.

          refs WT-1953
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/a07bd89a50813b9eb298edf6dd4f793b4cb40461

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: Only set the compaction flag once we find a page to compact. Otherwise files with a single page can end up having multiple checkpoints run on them. refs WT-1953 Branch: develop https://github.com/wiredtiger/wiredtiger/commit/a07bd89a50813b9eb298edf6dd4f793b4cb40461

            People

            • Assignee:
              keith.bostic Keith Bostic
              Reporter:
              kamran.khan Kamran K. (Inactive)
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: