-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Query Optimization
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The pipeline dependency graph code tracks field provenance as well as:
- missing fields
- constants from ExpressionConstant
- arrayness from PathArrayness API
In the future, we may want to add type inference and other field-level properties.
This ticket is about refactoring the existing field-level properties, so they are owned by policies.
We saw need for that in SKUNK-32.
For illustration, a refactor of the constant propagation out from the graph code into a policy:
class ConstantPolicy : public FieldPolicy { public: boost::optional<Value> get(FieldId id) const { auto it = _constants.find(id); ... } void onModify(FieldId id, const ModifyPath& path) { if (const auto* expr = path.getExpression().get()) { if (auto* c = dynamic_cast<const ExpressionConstant*>(expr)) { _constants[id] = c->getValue(); } } } void onRename(FieldId id, const RenamePath& path, FieldId sourceField, FieldMatchType sourceMatch) { switch (sourceMatch) { case FieldMatchType::kExact: if (auto c = get(sourceField)) { _constants[id] = *c; } break; case FieldMatchType::kShadowed: ... break; default: break; } } void onInclude(FieldId id, FieldId sourceField) { if (sourceField) { if (auto c = get(sourceField)) { _constants[id] = *c; } } } void onInvalidate(FieldId firstInvalid) { absl::erase_if(_constants, [firstInvalid](const auto& e) { return e.first >= firstInvalid; }); } private: absl::flat_hash_map<FieldId, Value> _constants; };
This will make the graph more composable and extendable. The graph can then be constructed with a bundle of policies specifying the required level of analysis (constants, arrayness, type information etc).
The graph code which currently does all of the above can be replaced with hooks into a bundle of policies, each of the policies can carry a mapping of FieldId -> custom property (some policies may not require it or can use special containers - bitsets and so on, since the FieldIds are incremental).
The policy methods can accept a Policies object with policies.get<ConstantPolicy>() for cross-policy interactions (if the type inference wants to use constant propgation information).