Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-38801

Cope with large BSON blobs in queryable's `listDirectory` call.

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.4.18, 3.6.9, 4.0.5
    • Fix Version/s: 3.6.11, 4.0.7, 4.1.8
    • Component/s: Storage
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Backport Requested:
      v4.0, v3.6, v3.4
    • Sprint:
      Storage NYC 2019-01-28

      Description

      When a (queryable) storage engine first starts up, it typically has to ask the operating system "What files are in my dbpath?"

      Queryable does this via a REST endpoint that returns a single BSON encoded payload that contains a list of file objects (filename, filesize, blocksize object/tuple). It is possible to have enough files to generate documents exceeding 50MB in size.

      Because queryable is only using this information to populate internal structures and never persisting that BSON object to disk, it seems plausible to relax the limits (as opposed to implementing a batching REST API to get the files in chunks, or circumventing the existing BSON parsing and hand-building a SAX-style parser).

      How BSON size limits are codified has changed across versions. I expect backports will need some massaging.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              daniel.gottlieb Daniel Gottlieb
              Reporter:
              daniel.gottlieb Daniel Gottlieb
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: