[COMPASS-3827] Import CSV Created: 16/Aug/19  Updated: 29/Oct/23  Resolved: 16/Oct/19

Status: Closed
Project: Compass
Component/s: Import/Export
Affects Version/s: None
Fix Version/s: 1.21.0

Type: Story Priority: Major - P3
Reporter: Massimiliano Marcon Assignee: Lucas Hrabovsky (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by COMPASS-3047 As a developer, I need to import CSV ... Closed
Epic Link: COMPASS-3013
Sprint: Iteration Ostrich

 Description   

User story

As a Compass user
I want to import documents from a CSV file
So that I can quickly seed my collection with existing data that may come from non-MongoDB tools

Acceptance Criteria

Notes

  • Lucas at some point suggested a CSV parser better than what Compass uses now (COMPASS-3129)
  • Note that selecting fields and their types is not part of this story


 Comments   
Comment by Githook User [ 30/Oct/19 ]

Author:

{'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com', 'name': 'Lucas Hrabovsky'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 29/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 23/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Generated at Wed Feb 07 22:34:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.