-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Schema
-
None
The schema analysis in mongodb-schema is already set up to take an async iterable. The Compass side however currently performs a toArray when doing the sample:
https://github.com/mongodb-js/compass/blob/d6b51a751f80c29be5c2d3fc579ed4711a1cb87d/packages/compass-schema/src/modules/schema-analysis.ts#L53
https://github.com/mongodb-js/compass/blob/d6b51a751f80c29be5c2d3fc579ed4711a1cb87d/packages/data-service/src/data-service.ts#L2255
I'm thinking in general we probably don't want to be doing that .toArray there and instead leave it up for consumers to do that. The sample is used in a couple other places we'll have to update if we do that, so maybe lets just do this one if that's a good amount of work.
Simon: The overall goal is to make the schema analysis algorithm smarter, by using a cursor and only fetching a batch of documents at once (e.g. 10), processing those, and then going to the next batch. That way, we only ever have to keep 10 documents in memory at a time, instead of 1000. Currently, collections with bloated dos cause Chrome to become unresponsive.
- is depended on by
-
COMPASS-9315 Stream documents from the sample cursor to the schema analysis
-
- In Code Review
-
- is related to
-
COMPASS-8925 Pass cancellable signals to driver queries
-
- Needs Triage
-