Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-31755

Raise intermediate $lookup document size to 100MB, and make it configurable

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.6.11, 4.0.6, 4.1.7
    • Component/s: None
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.0, v3.6
    • Sprint:
      Query 2018-12-17, Query 2018-12-31
    • Case:

      Description

      Issue Status as of Jan 24, 2019

      FEATURE DESCRIPTION

      The maximum size of a document during an intermediate $lookup stage will now be configurable with a setParameter called internalLookupStageIntermediateDocumentMaxSizeBytes. The default value is 100 MB. This parameter has a minimum value of 16 MB, the maximum document size allowed for a document being returned to the user.

      RATIONALE

      In general, documents are allowed to grow beyond the 16MB limit as they move through the pipeline--as long as the final documents which we serialize and send over the wire are within the limit.

      OPERATION

      During runtime, this parameter can be set on each mongod using the following command:

      db.adminCommand({setParameter: 1, internalLookupStageIntermediateDocumentMaxSizeBytes: <long>})
      

      On startup, each mongod can set this parameter with the following flag:

      --setParameter internalLookupStageIntermediateDocumentMaxSizeBytes=<long>
      

      To persist this setting in the configuration file, use the the setParameter option:

      setParameter:
         internalLookupStageIntermediateDocumentMaxSizeBytes: <long>
      

      FIX VERSIONS

      This feature will be included in MongoDB 3.6.11 and MongoDB 4.0.6.

      Original description

      In 3.4.7 $lookup appears to have an intermediate 16MB document limit. An aggregation that has a $lookup as an intermediate stage returned this error https://github.com/mongodb/mongo/blob/r3.4.7/src/mongo/db/pipeline/document_source_lookup.cpp#L159-L163.

      To address this a new setParameter,internalLookupStageIntermediateDocumentMaxSizeBytes, has been added to control the size of an intermediate $lookup stage. 

       

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: