Issue

On a sharded collection, the ShardedPartitioner will retrieve all chunks and create partitions (x chunks / partition). Then Spark executes all tasks in partition order, which means the n first queries (n = number of executors) will perform queries relative to the first n partitions, and so only the first shard will work until all chunks of the first shard have been queried, then second shard...

Which means we are only using a shard at a time :

Improvement proposition

Add shuffling support as an option to the ShardedPartitioner so the Spark tasks are executed randomly which balance the load across shards in case the data is sorted across all shards.

Example of implementation

Before the return of the method ShardedPartitioner.generatePartitions(...) :

if (readConfig.doShuffle()) {
    Collections.shuffle(partitions);
    return partitions;
}

With a new configuration option in the readConfig