Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38801

Cope with large BSON blobs in queryable's `listDirectory` call.

    XMLWordPrintableJSON

Details

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major - P3 Major - P3
    • 3.6.11, 4.0.7, 4.1.8
    • 3.4.18, 3.6.9, 4.0.5
    • Storage
    • None
    • Fully Compatible
    • v4.0, v3.6, v3.4
    • Storage NYC 2019-01-28

    Description

      When a (queryable) storage engine first starts up, it typically has to ask the operating system "What files are in my dbpath?"

      Queryable does this via a REST endpoint that returns a single BSON encoded payload that contains a list of file objects (filename, filesize, blocksize object/tuple). It is possible to have enough files to generate documents exceeding 50MB in size.

      Because queryable is only using this information to populate internal structures and never persisting that BSON object to disk, it seems plausible to relax the limits (as opposed to implementing a batching REST API to get the files in chunks, or circumventing the existing BSON parsing and hand-building a SAX-style parser).

      How BSON size limits are codified has changed across versions. I expect backports will need some massaging.

      Attachments

        Activity

          People

            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: