Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-383

Ensure partitioners handle empty collections

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 10.1.0
    • Affects Version/s: 10.0.5
    • Component/s: None
    • None

      The partitionner fails when manipulating an empty collection:

       

       

      Caused by: org.bson.BsonInvalidOperationException: Document does not contain key avgObjSize
    at org.bson.BsonDocument.throwIfKeyAbsent(BsonDocument.java:884)
    at org.bson.BsonDocument.getNumber(BsonDocument.java:178)
    at com.mongodb.spark.sql.connector.read.partitioner.SamplePartitioner.generatePartitions(SamplePartitioner.java:115)
    at com.mongodb.spark.sql.connector.read.partitioner.PartitionerHelper.generatePartitions(PartitionerHelper.java:54)

       

       

      The same problem existed and was solved on the 3.x connector, but the correction seems to be missing on the 10.x

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            cedric.vaneetvelde@soprabanking.com Cedric van Eetvelde
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: