ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Block Cache
    • None
    • Storage Engines - Persistence
    • 381.777
    • StorEng - Defined Pipeline
    • None

      layering follow up to WT-18794

      Context

      WT-18794 gates the victim cache so encrypted tables are skipped, because eviction caches the plaintext image and the read path then panics. That gate is a workaround, and the discussion on it was that this looks like a layering problem and shouldn't end there. This ticket is to look at it properly.

      The problem

      • PALite treats pages as opaque blobs, so it doesnt care whether a page is encrypted. The fix belongs on the WT side.
      • WT's eviction path (__evict_page_victim_cache) builds the cache image itself. It compresses and checksums but doesnt encrypt (comments in the function say "Not encrypted in this path"), so it hands back a different format from a normal put.
      • On read, __wt_blkcache_read_multi checks the page header flag and, if the btree has an encryptor and the flag is missing, panics: "multi_read unencrypted block for which encryption configured".
      • A cache hit through plh_get goes through the normal read path, and nothing tells that path the image came from a cache.

      Related existing behaviour

      WT's block cache already makes this choice explicitly ({}{{{}__blkcache_cache_wants_encrypted_data, block_io.c{}}}): RAM only caches store plaintext, caches that can hit external media store encrypted pages. The victim cache has no equivalent rule.

      Things to look into

      • Who should own encryption for cached images? Options: encrypt in eviction before plh_cache_put (mirroring __wt_blkcache_write), or have the read path know a cached image is already plaintext.
      • If we encrypt before caching, it needs more than setting the flag. The image needs the length field and ciphertext from wt_encrypt __ with the right skip, WT_PAGE_ENCRYPTED and WT_BLOCK_DISAGG_ENCRYPTED set, and the checksum rule fixed (CKSUM_UNENCRYPTED currently falls through). Setting the flag on a plaintext image would make __ _wt_decrypt read page bytes as a length field, which looks worse than the panic.
      • Check the keyed encryptor at read time is the one used at put time.
      • Cost: an extra encrypt pass on every clean eviction.
      • Whether the production PALI cache has any expectation about encrypted entries 

      Repro

      Only reproducible on the enable PALite victim cache in test/format branch. Remove the existing gate against encryption in config_disagg_victim_cache, then:

      ./t -c ../../../test/format/CONFIG.disagg runs.rows=400000 runs.ops=1500000 runs.timer=5 cache=64 disagg.victim_cache=1 disk.encryption=rotn-7
      
      file:WiredTigerShared.wt_stable, WT_CURSOR.insert: __blkcache_read_corrupt:
      fatal read error: multi_read unencrypted block for which encryption configured the process must exit
      
      • seeds that hit it: random.data_seed=9734762, random.extra_seed=2496206
      • not deterministic. 1 in 6 parallel runs hit it locally (3 more ran 420s clean, 2 hit an unrelated early exit at startup with cache=64), so it needs a few attempts.
         

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Mariam Mojid
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: