Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2670

Inefficient I/O when read full DB (poor readahead)

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.1, 3.2.12, 3.4.2, 3.5.1
    • Affects Version/s: WT2.8.0
    • Component/s: None
    • None
    • Storage 2016-10-31, Storage 2016-11-21, Storage 2016-12-12

      We see inefficiency in disk I/O when read full database (either by primary or secondary key). Our investigation shows that WT opens database and immediately sets POSIX_FADV_RANDOM for db and index files. POSIX_FADV_WILLNEED is set only for a small number of pages (as we understand for internal pages only). A quick test with preloading an empty posix_faddvise() shows almost 2x speedup in full database read. For 3.9 GB database stored on spinning disk we have:

      # regular posix_faddvise():
      PK total time: 45.1785 seconds, 17530462 records
      # empty posix_faddvise():
      PK total time: 25.8627 seconds, 17530462 records

      It is a common scenario for us to do full database scan. Is it possible to optimize this use case?

            sue.loverso@mongodb.com Susan LoVerso
            eiva Eugene Ivanov
            0 Vote for this issue
            16 Start watching this issue