Uploaded image for project: 'Compass '
  1. Compass
  2. COMPASS-9150

Change schema sample to pass cursor to analysis, not full doc array

    • Type: Icon: Task Task
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Schema
    • None
    • 3
    • Iteration A (Apr 21 - May 5), Iteration B (May 5 - May 16)
    • None
    • Developer Tools

      The schema analysis in mongodb-schema is already set up to take an async iterable. The Compass side however currently performs a toArray when doing the sample:
      https://github.com/mongodb-js/compass/blob/d6b51a751f80c29be5c2d3fc579ed4711a1cb87d/packages/compass-schema/src/modules/schema-analysis.ts#L53
      https://github.com/mongodb-js/compass/blob/d6b51a751f80c29be5c2d3fc579ed4711a1cb87d/packages/data-service/src/data-service.ts#L2255

      I'm thinking in general we probably don't want to be doing that .toArray there and instead leave it up for consumers to do that. The sample is used in a couple other places we'll have to update if we do that, so maybe lets just do this one if that's a good amount of work.

      Simon: The overall goal is to make the schema analysis algorithm smarter, by using a cursor and only fetching a batch of documents at once (e.g. 10), processing those, and then going to the next batch. That way, we only ever have to keep 10 documents in memory at a time, instead of 1000. Currently, collections with bloated dos cause Chrome to become unresponsive.

            Assignee:
            rhys.howell@mongodb.com Rhys Howell
            Reporter:
            rhys.howell@mongodb.com Rhys Howell
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: