[SERVER-38801] Cope with large BSON blobs in queryable's `listDirectory` call. Created: 31/Dec/18 Updated: 29/Oct/23 Resolved: 24/Jan/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 3.4.18, 3.6.9, 4.0.5 |
| Fix Version/s: | 3.6.11, 4.0.7, 4.1.8 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Gottlieb (Inactive) | Assignee: | Daniel Gottlieb (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v4.0, v3.6, v3.4
|
||||||||||||||||
| Sprint: | Storage NYC 2019-01-28 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
When a (queryable) storage engine first starts up, it typically has to ask the operating system "What files are in my dbpath?" Queryable does this via a REST endpoint that returns a single BSON encoded payload that contains a list of file objects (filename, filesize, blocksize object/tuple). It is possible to have enough files to generate documents exceeding 50MB in size. Because queryable is only using this information to populate internal structures and never persisting that BSON object to disk, it seems plausible to relax the limits (as opposed to implementing a batching REST API to get the files in chunks, or circumventing the existing BSON parsing and hand-building a SAX-style parser). How BSON size limits are codified has changed across versions. I expect backports will need some massaging. |
| Comments |
| Comment by Githook User [ 15/Feb/19 ] |
|
Author: {'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}Message: (cherry picked from commit 378d36eb24d3f93537692434ce1a6c9d87e6c7f3) |
| Comment by Githook User [ 06/Feb/19 ] |
|
Author: {'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}Message: (cherry picked from commit 378d36eb24d3f93537692434ce1a6c9d87e6c7f3) |
| Comment by Githook User [ 24/Jan/19 ] |
|
Author: {'username': 'dgottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb'}Message: |