Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38801

Cope with large BSON blobs in queryable's `listDirectory` call.

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.6.11, 4.0.7, 4.1.8
    • Affects Version/s: 3.4.18, 3.6.9, 4.0.5
    • Component/s: Storage
    • Labels:
    • Fully Compatible
    • v4.0, v3.6, v3.4
    • Storage NYC 2019-01-28

      When a (queryable) storage engine first starts up, it typically has to ask the operating system "What files are in my dbpath?"

      Queryable does this via a REST endpoint that returns a single BSON encoded payload that contains a list of file objects (filename, filesize, blocksize object/tuple). It is possible to have enough files to generate documents exceeding 50MB in size.

      Because queryable is only using this information to populate internal structures and never persisting that BSON object to disk, it seems plausible to relax the limits (as opposed to implementing a batching REST API to get the files in chunks, or circumventing the existing BSON parsing and hand-building a SAX-style parser).

      How BSON size limits are codified has changed across versions. I expect backports will need some massaging.

            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            0 Vote for this issue
            10 Start watching this issue