Major - P3 FEATURE DESCRIPTION
The maximum size of a document during an intermediate $lookup stage will now be configurable with a setParameter called internalLookupStageIntermediateDocumentMaxSizeBytes. The default value is 100 MB. This parameter has a minimum value of 16 MB, the maximum document size allowed for a document being returned to the user.
RATIONALE
In general, documents are allowed to grow beyond the 16MB limit as they move through the pipeline--as long as the final documents which we serialize and send over the wire are within the limit.
OPERATION
During runtime, this parameter can be set on each mongod using the following command:
db.adminCommand({setParameter: 1, internalLookupStageIntermediateDocumentMaxSizeBytes: <long>})
On startup, each mongod can set this parameter with the following flag:
--setParameter internalLookupStageIntermediateDocumentMaxSizeBytes=<long>
To persist this setting in the configuration file, use the the setParameter option:
setParameter: internalLookupStageIntermediateDocumentMaxSizeBytes: <long>
FIX VERSIONS
This feature will be included in MongoDB 3.6.11 and MongoDB 4.0.6.
Original description
In 3.4.7 $lookup appears to have an intermediate 16MB document limit. An aggregation that has a $lookup as an intermediate stage returned this error https://github.com/mongodb/mongo/blob/r3.4.7/src/mongo/db/pipeline/document_source_lookup.cpp#L159-L163.
To address this a new setParameter,internalLookupStageIntermediateDocumentMaxSizeBytes, has been added to control the size of an intermediate $lookup stage.