-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Block Cache
-
None
-
Storage Engines - Persistence
-
381.777
-
StorEng - Defined Pipeline
-
None
layering follow up to WT-18794
Context
WT-18794 gates the victim cache so encrypted tables are skipped, because eviction caches the plaintext image and the read path then panics. That gate is a workaround, and the discussion on it was that this looks like a layering problem and shouldn't end there. This ticket is to look at it properly.
The problem
- PALite treats pages as opaque blobs, so it doesnt care whether a page is encrypted. The fix belongs on the WT side.
- WT's eviction path (__evict_page_victim_cache) builds the cache image itself. It compresses and checksums but doesnt encrypt (comments in the function say "Not encrypted in this path"), so it hands back a different format from a normal put.
- On read, __wt_blkcache_read_multi checks the page header flag and, if the btree has an encryptor and the flag is missing, panics: "multi_read unencrypted block for which encryption configured".
- A cache hit through plh_get goes through the normal read path, and nothing tells that path the image came from a cache.
Related existing behaviour
WT's block cache already makes this choice explicitly ({}{{{}__blkcache_cache_wants_encrypted_data, block_io.c{}}}): RAM only caches store plaintext, caches that can hit external media store encrypted pages. The victim cache has no equivalent rule.
Things to look into
- Who should own encryption for cached images? Options: encrypt in eviction before plh_cache_put (mirroring __wt_blkcache_write), or have the read path know a cached image is already plaintext.
- If we encrypt before caching, it needs more than setting the flag. The image needs the length field and ciphertext from wt_encrypt __ with the right skip, WT_PAGE_ENCRYPTED and WT_BLOCK_DISAGG_ENCRYPTED set, and the checksum rule fixed (CKSUM_UNENCRYPTED currently falls through). Setting the flag on a plaintext image would make __ _wt_decrypt read page bytes as a length field, which looks worse than the panic.
- Check the keyed encryptor at read time is the one used at put time.
- Cost: an extra encrypt pass on every clean eviction.
- Whether the production PALI cache has any expectation about encrypted entries
Repro
Only reproducible on the enable PALite victim cache in test/format branch. Remove the existing gate against encryption in config_disagg_victim_cache, then:
./t -c ../../../test/format/CONFIG.disagg runs.rows=400000 runs.ops=1500000 runs.timer=5 cache=64 disagg.victim_cache=1 disk.encryption=rotn-7
file:WiredTigerShared.wt_stable, WT_CURSOR.insert: __blkcache_read_corrupt: fatal read error: multi_read unencrypted block for which encryption configured the process must exit
- seeds that hit it: random.data_seed=9734762, random.extra_seed=2496206
- not deterministic. 1 in 6 parallel runs hit it locally (3 more ran 420s clean, 2 hit an unrelated early exit at startup with cache=64), so it needs a few attempts.