Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-254

Fields values in Mongodb logs are hidden when inserting from Robo3T

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 2.1.6
    • Fix Version/s: None
    • Component/s: Logging, Writes
    • Labels:

      Description

      Spark connector is inserting documents to MongoDB, with following configuration. Trying to capture the error logs when Duplicates getting inserted into Mongo.

       

      Spark Connector inserting to MongoDB Sharded cluster, Uniqueness on Shard key

       

      In the mongod logs, shard key field values are hiding, so I am unable to capture which is getting duplicated.

       

      There is no visibility from Spark Connector to capture Logs

       

      Spark Connector code:

      object Write2Mongo {
        def main(args: Array[String]): Unit = {
          if(args.length != 2)

      { _println_("Usage: <CONF FILE> <STG DIR>"); sys._exit_(1) }

          val confFile = args(0)
          val stgDir = args(1)

          /**************************************************
            * Build Spark context
            **************************************************/
          __     val config = ConfigFactory.parseFile(new File(confFile))
          val spark = MongoContext.buildSparkContext(config)
          val mongoRdd = spark.read.parquet(stgDir).rdd.map(x => convertBytes2Bson(x.getAs[Array[Byte]]("document")))
          MongoSpark.save(mongoRdd)
        }
      }

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ross.lawley Ross Lawley
              Reporter:
              gundadheeraj8@gmail.com Dheeraj Gunda
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: