Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2035

index read leads to read of main table

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: WT2.6.1
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      10
    • Last comment by Customer:
      true

      Description

      Hi!

      We have a table with index and sometimes want to read only index.
      I.e. we're only interested in pairs (index key, primary key)

      So we tried to use index cursor projection as described here

      But it seems that WiredTiger nevertheless reads the main table although it doesn't need to (I think).

      The problem is that our main table is quite big (several GB's) and scanning through index lead to it being read non-sequentially which is quite slow.

      Small demo example attached (should be build with -rdynamic)
      Output I get:

      $ ./read_index
      Filling DB...
      total 481716
      -rw-r--r-- 1 sbn eng    757760 Aug  7 15:03 main_index.wti
      -rw-r--r-- 1 sbn eng 491974656 Aug  7 15:03 main.wt
      -rw-r--r-- 1 sbn eng        45 Aug  7 15:03 WiredTiger
      -rw-r--r-- 1 sbn eng       298 Aug  7 15:03 WiredTiger.basecfg
      -rw-r--r-- 1 sbn eng        21 Aug  7 15:03 WiredTiger.lock
      -rw-r--r-- 1 sbn eng       938 Aug  7 15:03 WiredTiger.turtle
      -rw-r--r-- 1 sbn eng     36864 Aug  7 15:03 WiredTiger.wt
       
      Reading index using index: cursor...
      total 0
      lrwx------ 1 sbn eng 64 Aug  7 15:03 0 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 1 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 2 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 3 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/WiredTiger.lock
      lrwx------ 1 sbn eng 64 Aug  7 15:03 4 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/WiredTiger.wt
      lrwx------ 1 sbn eng 64 Aug  7 15:03 5 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/main_index.wti
      lrwx------ 1 sbn eng 64 Aug  7 15:03 6 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/main.wt
      key_format = i, value_format = i
      pread() fd stats:
      5:      745472
      6:      491962368
       
      Reading index using file: cursor...
      total 0
      lrwx------ 1 sbn eng 64 Aug  7 15:03 0 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 1 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 2 -> /dev/pts/2
      lrwx------ 1 sbn eng 64 Aug  7 15:03 3 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/WiredTiger.lock
      lrwx------ 1 sbn eng 64 Aug  7 15:03 4 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/WiredTiger.wt
      lrwx------ 1 sbn eng 64 Aug  7 15:03 5 -> /home/sbn/src/wiredtiger/examples/c/read_index.db/main_index.wti
      key_format = ii, value_format = 
      pread() fd stats:
      5:      745472
      

      Thanks!

      1. read_index.c
        3 kB
        Dmitri Shubin

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2171 from wiredtiger/wt-2035-index-read

          WT-2035. For index cursors, keep track of which column groups need to be positioned.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/aee1c9406a13f5f1a2bc0a950ebf621407afe4b0

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2171 from wiredtiger/wt-2035-index-read WT-2035 . For index cursors, keep track of which column groups need to be positioned. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/aee1c9406a13f5f1a2bc0a950ebf621407afe4b0
          Hide
          donald.anderson Donald Anderson added a comment -

          Dmitri Shubin, The fix merged into WT's develop branch should fix the original problem. Thanks for the concise bug report!

          Show
          donald.anderson Donald Anderson added a comment - Dmitri Shubin , The fix merged into WT's develop branch should fix the original problem. Thanks for the concise bug report!
          Hide
          Dmitri Shubin Dmitri Shubin added a comment -

          Donald Anderson, thank you!
          We checked 'develop' branch and it works fine for us (both with and without custom extractor).

          Are there plans to release 2.6.2 any time soon?

          Show
          Dmitri Shubin Dmitri Shubin added a comment - Donald Anderson , thank you! We checked 'develop' branch and it works fine for us (both with and without custom extractor). Are there plans to release 2.6.2 any time soon?
          Hide
          michael.cahill Michael Cahill added a comment -

          Dmitri Shubin, I can't give you a date, but we do plan to cut a new WT release soon.

          Show
          michael.cahill Michael Cahill added a comment - Dmitri Shubin , I can't give you a date, but we do plan to cut a new WT release soon.
          Hide
          Dmitri Shubin Dmitri Shubin added a comment -

          Michael Cahill, thanks!

          It would be great for us to have it by the end of September.
          Or probably we can just cherry-pick some commits?

          Show
          Dmitri Shubin Dmitri Shubin added a comment - Michael Cahill , thanks! It would be great for us to have it by the end of September. Or probably we can just cherry-pick some commits?

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 32 weeks, 6 days ago
                Date of 1st Reply: