[COMPASS-3853] Replace backend of the feature with better CSV/JSON parsers Created: 28/Aug/19  Updated: 29/Oct/23  Resolved: 01/Oct/19

Status: Closed
Project: Compass
Component/s: Import/Export
Affects Version/s: None
Fix Version/s: 1.21.0

Type: Task Priority: Major - P3
Reporter: Massimiliano Marcon Assignee: Lucas Hrabovsky (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by COMPASS-3864 [Spike] Can we use CSV and JSON modul... Closed
Epic Link: COMPASS-3013
Story Points: 5
Sprint: Iteration Manatee, Iteration Newt

 Description   

This task is about replacing the backend of the feature with community-maintained libraries that are battle-tested for CSV and JSON parsing/creation.

This is timeboxed to 1 week and the main goal is that after this is done, Import/Export in Compass with the existing UX works reliably, no matter the size of the collection:

  • CSV are exported according to the CSV spec and can be imported back
  • JSON is imported with no errors. For the time being, since we don't change the UX, it's ok to still only support JSONL but let's keep in mind that eventually we want to also support JSON arrays. JSON is also exported with no errors, and an exported JSON can be imported back.

If we are unable to make this work, we will fallback to using mongoimport/mongoexport.



 Comments   
Comment by Githook User [ 30/Oct/19 ]

Author:

{'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com', 'name': 'Lucas Hrabovsky'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 29/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 23/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Generated at Wed Feb 07 22:34:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.