Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-97113

Fine-grained dependency-based query cache invalidation

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization

      The ultimate goal of this ticket is to reduce the number of queries that need to be multi-planned. Currently any change to a collection indexes causes query cache invalidation. As a result, this may cause unnecessary re-planning (currently via multi-planning) of a large number of queries.

      In practice it may very well be the case that a change of a certain index affects a very small number of queries, and therefore all this re-planning was in vain. This can be achieved by maintaining a set of fields for each query that can potentially be affected by a change in available indexes. A simplistic approach could just maintain all query fields. A more advanced approach could maintain a subset of all query fields that could use an index. For instance, these would be the fields of all sargable predicates, all sort and group fields, and the join field(s) of a $lookup. Let's call this set "sensitive fields".

      When an index is modified (dropped or created), the query cache invalidation logic checks the intersection of the index's fields with the set of sensitive fields of each query. If the intersection is empty, the query is not affected by the change, and doesn't need to be re-planned.

      This check must be performed very efficiently in order to improve re-planning overhead. It is well known that set operations can be implemented very efficiently via bitsets. Therefore this task requires to:

      • discover the sensitive fields at query level
      • enumerate all fields of all queries (ideally densely)
      • when creating indexes, find the IDs of the index's fields among the set of sensitive fields (e.g. using a hash table)

      Then we can efficiently check index <-> query dependencies.

      •  

            Assignee:
            Unassigned Unassigned
            Reporter:
            timour.katchaounov@mongodb.com Timour Katchaounov
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: