-
Type: Improvement
-
Resolution: Won't Fix
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
If I use Mongo Spark builtin UDF to use ObjectId in a where clause, the condition does not get pushed down to RDD filters:
Example query:
SELECT * FROM customers WHERE _id = ObjectId('4f398939e4b0d1716f5a7e2f')
If I check it with explain:
EXPLAIN SELECT * FROM customers WHERE _id = ObjectId('4f398939e4b0d1716f5a7e2f')
I see following along the lines:
PushedFilters: [IsNotNull(_id)]
This is because the UDF does not implement foldable and therefore cannot be statically calculated and pushed to the PushedFilters. ScalaUDF extends Expression which has the method foldable which explains it a bit.