Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-239

Single field nested column with null value fails on save

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.4.1, 2.3.3, 2.2.7, 2.1.6
    • Affects Version/s: 2.4.0
    • Component/s: Writes
    • None
    • Environment:
      Databricks

      Updated

      Support disabling bson helpers

      Was

      Writing a record to MongoDb from Databricks spark dataframe fails in a peculiar manner related to a null value in a nested column that has only a single value. Looking at the stack trace, it appears that the javascript codec gets chosen for nested structures that have only a single value. In turn, bson.AbstractBsonWriter.writeJavaScript will not accept null values and so fails.

      If you look at the attached image, you will see nested, with only a single field, fails, whereas nested 2 succeeds.

      Larger image along with full stack trace attached.

        1. image-2019-03-28-16-42-10-488.png
          image-2019-03-28-16-42-10-488.png
          6 kB
        2. image-2019-03-29-11-10-55-882.png
          image-2019-03-29-11-10-55-882.png
          41 kB
        3. mongbugstacktrace.txt
          20 kB
        4. mongobug.png
          mongobug.png
          293 kB

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            dbranscombe Dwight Branscombe
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: