Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-68139

Resharding command fails if the projection sort is bigger than 100MB

    • Fully Compatible
    • v6.1, v6.0, v5.0
    • Sharding 2022-09-05, Sharding 2022-09-19, Sharding 2022-10-03
    • 3

      Problem: The resharding command fails if the newShardKeyProjection size * samplePerChunk * intialNumOfChunks is bigger than 100MB.

      newShardKeyProjection: size of newShardKey document

      Default value of initialNumOfChunks is the existing number of chunks before the resharding.

      Default value of samplePerChunk: 10

      Description: Resharding creates a pipeline query command and the query is then sorted on the new Shard Key as part of the projection. If the projection of over 100MB is passed onto sort, the following error is thrown 

      "errmsg" : "PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.", 

      The error is due to 
      ReshardingSplitPolicy::_makePipelineDocumentSource creating ExpressionContext with allowDiskUse to false. 

      auto expCtx = make_intrusive<ExpressionContext>(opCtx,                                                    boost::none, /* explain */                                                    false,       /* fromMongos */                                                    false,       /* needsMerge */                                                    false,       /* allowDiskUse */                                                    true,        /* bypassDocumentValidation */                                                    false,       /* isMapReduceCommand */                                                    ns,                                                    boost::none, /* runtimeConstants */                                                    nullptr,     /* collator */                                                    std::move(pi),                                                    std::move(resolvedNamespaces),                                                    boost::none); /* collUUID */ 

      Steps to reproduce:
       Attached .js test and initial_split_policy.h to change the kDefaultSamplesPerChunk to 10000(from 10)



        1. initial_split_policy.h
          15 kB
        2. reshard_coll1.js
          2 kB

            nandini.bhartiya@mongodb.com Nandini Bhartiya
            rachita.dhawan@mongodb.com Rachita Dhawan
            1 Vote for this issue
            7 Start watching this issue