[COMPASS-3829] Import JSON Created: 16/Aug/19  Updated: 29/Oct/23  Resolved: 23/Oct/19

Status: Closed
Project: Compass
Component/s: Import/Export
Affects Version/s: None
Fix Version/s: 1.21.0

Type: Story Priority: Major - P3
Reporter: Massimiliano Marcon Assignee: Lucas Hrabovsky (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by COMPASS-3185 As a frequent user, I need to import ... Closed
is caused by COMPASS-3384 As a developer I need Compass to acce... Closed
is caused by COMPASS-3755 Improve importing invalid JSON Closed
Epic Link: COMPASS-3013
Story Points: 3
Sprint: Iteration Ostrich, Iteration Python

 Description   

User story

As a Compass user
I want to import documents from a JSON file
So that I can quickly seed my collection with existing data

Acceptance Criteria

  • Compass can import a valid JSON file (an array of documents, COMPASS-3185)
  • Compass cam import extended JSON (COMPASS-3384)
  • Compass can import a valid JSONL file (http://jsonlines.org/)
  • When an error happens during the import
    • Compass shows a meaningful error message
    • Compass says how many documents it managed to import
  • When the JSON is invalid
  • During the import, progress is displayed in the import modal

Notes

  • JSONL = whatever is the spec that mongoimport handles


 Comments   
Comment by Githook User [ 30/Oct/19 ]

Author:

{'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com', 'name': 'Lucas Hrabovsky'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 29/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 23/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Generated at Wed Feb 07 22:34:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.