[COMPASS-6528] Consistent progress support for analyzeCSVFields, importCSV and importJSON, etc Created: 17/Feb/23  Updated: 23/Feb/23  Resolved: 23/Feb/23

Status: Closed
Project: Compass
Component/s: None
Affects Version/s: None
Fix Version/s: No version

Type: Task Priority: Major - P3
Reporter: Le Roux Bodenstein Assignee: Le Roux Bodenstein
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Link: COMPASS-5576
Story Points: 2
Documentation Changes: Not Needed
Sprint: Iteration Velvet Crab

 Description   

For importing CSV I think the progress event worked fine because you could see how many docs have been processed and an earlier step already returned the number of docs when we analysed the file. This is enough to accurately show the progress as designed.

For importing JSON we don't know how many docs there are in advance.

So I'll have to augment the progress callback with how many bytes have been processed. Then the progress can be the number of bytes out of the total.Add it to both importCSV() and importJSON() so they can remain compatible.

ie. you would have the number of docs and number of bytes processed. Total bytes in the file can be read outside the function and for CSV we have the additional advantage that we already have the total number of docs.

Also check other import/export functions that have a progress callback.



 Comments   
Comment by Githook User [ 22/Feb/23 ]

Author:

{'name': 'Le Roux Bodenstein', 'email': 'lerouxb@gmail.com', 'username': 'lerouxb'}

Message: feat(import): progressCallback(numDocs, numBytes) COMPASS-6528 (#4080)

  • progressCallback(numDocs, numBytes)
  • fix error handling
Comment by Githook User [ 22/Feb/23 ]

Author:

{'name': 'Le Roux Bodenstein', 'email': 'lerouxb@gmail.com', 'username': 'lerouxb'}

Message: feat(import): progressCallback(numDocs, numBytes) COMPASS-6528 (#4080)

  • progressCallback(numDocs, numBytes)
  • fix error handling
Comment by Githook User [ 21/Feb/23 ]

Author:

{'name': 'Le Roux Bodenstein', 'email': 'lerouxb@gmail.com', 'username': 'lerouxb'}

Message: feat(import): progressCallback(numDocs, numBytes) COMPASS-6528 (#4080)

  • progressCallback(numDocs, numBytes)
  • fix error handling
Generated at Wed Feb 07 22:43:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.