Uploaded image for project: 'Compass '
  1. Compass
  2. COMPASS-6638

Limit the size/time that analyzeCSVFile() runs

    • Type: Icon: Task Task
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.36.2
    • Affects Version/s: None
    • Component/s: None
    • None
    • 2
    • Not Needed

      For some outlier sized files analyzeCSVFields() can take minutes to run. We can try and optimise it but there's a point where we might as well stop and treat the amount gathered so far as a sample.

      We can either set a time limit (ie. stop at 1 minute) or a size limit (either bytes or number of rows).

      Then we should probably show in the UI that it is based on a sample..

      In the stats we have files of that kind of size are exceedingly rare. But if you look at tickets filed in JIRA it seems that it does happen, probably just with the kinds of people or organisations that opted out of stats collection.

      In the end we went with a loader that has a skip button. The user can skip at any point and then it will go with what it detected up to that point.

            leroux.bodenstein@mongodb.com Le Roux Bodenstein
            leroux.bodenstein@mongodb.com Le Roux Bodenstein
            0 Vote for this issue
            0 Start watching this issue