Add shuffling support to partitions (ShardedPartitioner)

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Unknown
    • 11.2.0, 10.8.0
    • Affects Version/s: None
    • Component/s: None
    • Java Drivers
    • Needed
    • Hide

      1. What would you like to communicate to the user about this feature?

      This feature adds two new settings 

      name type description
      spark.mongodb.read.partitioner.options.shuffle  boolean When true, shuffles the generated partitions so consecutive partitions are unlikely to belong to the same shard, spreading read load across shards.
      spark.mongodb.read.partitioner.options.shuffle.seed long The seed used when shuffling partitions. Set it to get a reproducible shuffle. Only takes effect when shuffle is true. An invalid (non-long) value raises a MongoSparkException.
      Optional parameter, when not specified random seed will be used 

      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? This feature adds two new settings  name type description spark.mongodb.read.partitioner.options.shuffle  boolean When true, shuffles the generated partitions so consecutive partitions are unlikely to belong to the same shard, spreading read load across shards. spark.mongodb.read.partitioner.options.shuffle.seed long The seed used when shuffling partitions. Set it to get a reproducible shuffle. Only takes effect when shuffle is true. An invalid (non-long) value raises a MongoSparkException. Optional parameter, when not specified random seed will be used  2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Issue

      On a sharded collection, the ShardedPartitioner will retrieve all chunks and create partitions (x chunks / partition). Then Spark executes all tasks in partition order, which means the n first queries (n = number of executors) will perform queries relative to the first n partitions, and so only the first shard will work until all chunks of the first shard have been queried, then second shard...

      Which means we are only using a shard at a time :

      Improvement proposition

      Add shuffling support as an option to the ShardedPartitioner so the Spark tasks are executed randomly which balance the load across shards in case the data is sorted across all shards.

      Example of implementation

      Before the return of the method ShardedPartitioner.generatePartitions(...) :

      if (readConfig.doShuffle()) {
    Collections.shuffle(partitions);
    return partitions;
}

      With a new configuration option in the readConfig

        1. image-2026-06-19-17-49-05-104.png
          image-2026-06-19-17-49-05-104.png
          239 kB

            Assignee:
            Almas Abdrazak
            Reporter:
            Antonin LEGRAND (EXT)
            Ross Lawley
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: