Streaming Mode - Read - Configuration - Rate Limiting

XMLWordPrintableJSON

    • Java Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      I would greatly appreciate an option to limit the size of incoming streaming batches, similar to Kafka's maxOffsetsPerTrigger. This would greatly improve the utility of the Mongo Spark Connector for high-volume collections. At B.Well we currently have several collections wherein incoming batches are too large to ingest without using extremely large nodes which are very expensive to run. Although I'm aware we could use the `change.stream.micro.batch.max.partition.count` option, however this is sub-optimal for a number of reasons, starting with the out-of-order processing which would require us to introduce an additional hop (append-only streaming table) between the driver and our target tables in order to sort the changes by clusterTime prior to merging—we'd prefer to keep it simple and reduce overhead by simply merging directly into our replica dataset.

      Please drop me a line if you have any question or alternate recommendations—thanks! 

      cc:  seamus.noonan@mongodb.com 

            Assignee:
            Unassigned
            Reporter:
            David Belais
            None
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: