Resolution: Won't Fix
Affects Version/s: None
Fix Version/s: None
@michaelcahill, I took a closer look at the filter support today, and I have a couple of comments:
- We probably don't want to use the word "filter", that's a large concept/word to reserve for this functionality; I was thinking maybe "discard filters".
- If we implement this as we discussed (in Btree page reconciliation), then it only works for file: objects, because we don't have the primary key for anything other than the single object. I poked around a bit trying to think of a way to get my hands on a primary key down in reconciliation, but I don't see any obvious way to do that, even if it were done as a high-level cursor and a separate pass over the page.
- I think this conclusion applies to LSM as well?
- Does it make this feature sufficiently useless that we don't want to do the work?
- I think the handles probably look a lot like WT_COLLATORs, but some questions came up:
- It seems like discard filters would potentially be transient: if we set the filter value in the metadata, do we want to go to the effort of making utilities know about those filters, that is, is it a failure if we can't find the filter that was configured at object create, or do we simply ignore it?
- Would we want to support reconfiguration of the filter using WT_SESSION.reconfigure? If so, does reconfigure rewrite the metadata?
- Would we provide an external API for discard filters? It seems like the answer is yes, external data sources might want to implement the feature.
- We can get into trouble if a discard filter returns inconsistent results. For example, if we reconcile a page and the filter returns "delete key AAA", but the next time we reconcile the page it returns "keep key AAA", we're going to corrupt the underlying store, chunks of reconciliation depend on repeatable results. We can make this work, and I'm guessing we'd have to, but wanted to raise the question.