Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-68139

Resharding command fails if the projection sort is bigger than 100MB

    XMLWordPrintable

Details

    • Fully Compatible
    • v6.1, v6.0, v5.0
    • Sharding 2022-09-05, Sharding 2022-09-19, Sharding 2022-10-03
    • 3

    Description

      Problem: The resharding command fails if the newShardKeyProjection size * samplePerChunk * intialNumOfChunks is bigger than 100MB.

      newShardKeyProjection: size of newShardKey document

      Default value of initialNumOfChunks is the existing number of chunks before the resharding.

      Default value of samplePerChunk: 10

      Description: Resharding creates a pipeline query command and the query is then sorted on the new Shard Key as part of the projection. If the projection of over 100MB is passed onto sort, the following error is thrown 

      "errmsg" : "PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.", 

      The error is due to 
      ReshardingSplitPolicy::_makePipelineDocumentSource creating ExpressionContext with allowDiskUse to false. 

      auto expCtx = make_intrusive<ExpressionContext>(opCtx,                                                    boost::none, /* explain */                                                    false,       /* fromMongos */                                                    false,       /* needsMerge */                                                    false,       /* allowDiskUse */                                                    true,        /* bypassDocumentValidation */                                                    false,       /* isMapReduceCommand */                                                    ns,                                                    boost::none, /* runtimeConstants */                                                    nullptr,     /* collator */                                                    std::move(pi),                                                    std::move(resolvedNamespaces),                                                    boost::none); /* collUUID */ 

      Steps to reproduce:
       Attached .js test and initial_split_policy.h to change the kDefaultSamplesPerChunk to 10000(from 10)
       

       

       

      Attachments

        1. initial_split_policy.h
          15 kB
        2. reshard_coll1.js
          2 kB

        Issue Links

          Activity

            People

              nandini.bhartiya@mongodb.com Nandini Bhartiya
              rachita.dhawan@mongodb.com Rachita Dhawan
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: