As part of the query settings project we made QueryShapeHash computation "lazy", i.e. it is performed only when at least a single query setting on the collection is set.
We should do similar thing for index filters. This should give us performance benefits of not performing unnecessary computation.