Details
Description
Problem: The resharding command fails if the newShardKeyProjection size * samplePerChunk * intialNumOfChunks is bigger than 100MB.
newShardKeyProjection: size of newShardKey document
Default value of initialNumOfChunks is the existing number of chunks before the resharding.
Default value of samplePerChunk: 10
Description: Resharding creates a pipeline query command and the query is then sorted on the new Shard Key as part of the projection. If the projection of over 100MB is passed onto sort, the following error is thrown
"errmsg" : "PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.", |
The error is due to
ReshardingSplitPolicy::_makePipelineDocumentSource creating ExpressionContext with allowDiskUse to false.
auto expCtx = make_intrusive<ExpressionContext>(opCtx, boost::none, /* explain */ false, /* fromMongos */ false, /* needsMerge */ false, /* allowDiskUse */ true, /* bypassDocumentValidation */ false, /* isMapReduceCommand */ ns, boost::none, /* runtimeConstants */ nullptr, /* collator */ std::move(pi), std::move(resolvedNamespaces), boost::none); /* collUUID */ |
Steps to reproduce:
Attached .js test and initial_split_policy.h to change the kDefaultSamplesPerChunk to 10000(from 10)
Attachments
Issue Links
- related to
-
SERVER-70012 Getting 'Sort exceeded memory limit' error when trying to reshard a collection
-
- Closed
-