[COMPASS-994] Slow performance and UI unresponsive on collection with 1,000+ fields Created: 02/Apr/17  Updated: 27/Oct/23  Resolved: 01/Oct/18

Status: Closed
Project: Compass
Component/s: Performance, Schema
Affects Version/s: 1.6.1
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Sigfrido Narvaez Assignee: Unassigned
Resolution: Gone away Votes: 0
Labels: stagnant
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OSX 10.12.4 16GB RAM 15" Macbook Pro


Attachments: File COMPASS-994-with-37k-docs-test.mov     File IoTSchema1kfields.json    
Issue Links:
Related
related to COMPASS-983 Spike: Research schema view performance Closed
Epic Link: COMPASS-1739

 Description   

Received a proposed schema/dat set for an IoT use case from a prospect. The document has 1,000+ fields and 36,000 docs in the data set. When trying to load into Compass, the schema analysis takes several minutes, and when completed, Compass becomes unresponsive, and my CPU starts to max out.

Even though the schema could be changed to use nested documents and arrays (that will be advised to the prospect) - our Compass product should perform properly.



 Comments   
Comment by Durran Jordan [ 01/Oct/18 ]

Rewriting stories to remove hadron-document

Comment by Sam Weaver [ 20/Jul/17 ]

For comparison - Compass latest master takes 31 seconds to load the documents view (with 37,000 of these documents). Monogchef takes 6-8 seconds on the same collection.

Comment by Peter Schmidt [ 20/Jul/17 ]

I have retested this in v1.8.0-beta.2.

In isolation (one document) Compass behaves pretty responsively. However, with 37000 copies imported, using this SO to rename the _id field and avoid duplicate key errors, I get the following 3:22 long video:
COMPASS-994-with-37k-docs-test.mov

I think the Schema Analysis now behaves better than the Documents Tab.

There should at least be a loading spinner for the ~20 seconds Compass spends rendering the Documents Tab (assuming no simple optimisations like COMPASS-1323 exist) so the Compass user isn't left wondering whether Compass has hung while the Documents Tab is loading.

Comment by Thomas Rueckstiess [ 20/Apr/17 ]

This should be addressed when we finalize the epic COMPASS-507, which means the Documents tab will become the default, and the schema analysis does not run by default anymore, but only on demand. The user will also have the opportunity to change the limit to less documents for schema analysis.

Generated at Wed Feb 07 22:26:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.