[SERVER-38801] Cope with large BSON blobs in queryable's `listDirectory` call. Created: 31/Dec/18  Updated: 29/Oct/23  Resolved: 24/Jan/19

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 3.4.18, 3.6.9, 4.0.5
Fix Version/s: 3.6.11, 4.0.7, 4.1.8

Type: Improvement Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-36606 Remove size limits on BSON audit events Closed
Related
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.0, v3.6, v3.4
Sprint: Storage NYC 2019-01-28
Participants:

 Description   

When a (queryable) storage engine first starts up, it typically has to ask the operating system "What files are in my dbpath?"

Queryable does this via a REST endpoint that returns a single BSON encoded payload that contains a list of file objects (filename, filesize, blocksize object/tuple). It is possible to have enough files to generate documents exceeding 50MB in size.

Because queryable is only using this information to populate internal structures and never persisting that BSON object to disk, it seems plausible to relax the limits (as opposed to implementing a batching REST API to get the files in chunks, or circumventing the existing BSON parsing and hand-building a SAX-style parser).

How BSON size limits are codified has changed across versions. I expect backports will need some massaging.



 Comments   
Comment by Githook User [ 15/Feb/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-38801: Up the size limit for a listDirectory queryable API to the BufBuilder limit.

(cherry picked from commit 378d36eb24d3f93537692434ce1a6c9d87e6c7f3)
Branch: v3.6
https://github.com/10gen/mongo-enterprise-modules/commit/680a265d89a245b28bbf570ed7f6aeefe517fa9f

Comment by Githook User [ 06/Feb/19 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-38801: Up the size limit for a listDirectory queryable API to the BufBuilder limit.

(cherry picked from commit 378d36eb24d3f93537692434ce1a6c9d87e6c7f3)
Branch: v4.0
https://github.com/10gen/mongo-enterprise-modules/commit/d49f176ad29881c795d7179918d44c88799b1b3e

Comment by Githook User [ 24/Jan/19 ]

Author:

{'username': 'dgottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb'}

Message: SERVER-38801: Up the size limit for a listDirectory queryable API to the BufBuilder limit.
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/378d36eb24d3f93537692434ce1a6c9d87e6c7f3

Generated at Thu Feb 08 04:50:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.