[COMPASS-3828] Export JSON Created: 16/Aug/19  Updated: 29/Oct/23  Resolved: 16/Oct/19

Status: Closed
Project: Compass
Component/s: Import/Export
Affects Version/s: None
Fix Version/s: 1.21.0

Type: Story Priority: Major - P3
Reporter: Massimiliano Marcon Assignee: Lucas Hrabovsky (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Epic Link: COMPASS-3013
Sprint: Iteration Ostrich

 Description   

User story

As a Compass user
I want to export documents as a JSON
So that I use my data with other tools

Acceptance Criteria

  • Compass can export documents to a valid (extended) JSON file
  • When filter, project, sort or limit are specified for the collection the user is exporting, the user has the option to use them as the settings for the export (https://mongodb.invisionapp.com/share/BVNOZB4DYT3#/screens/347756471)
  • When an error happens during the export
    • Compass shows a meaningful error message
    • Compass says how many documents it managed to export
  • During the export, progress is displayed in the export modal

Notes

  • Note that I am suggesting we export valid JSON instead of JSON lines. Valid JSON makes more sense because it can be used also with tools outside the MongoDB ecosystem
  • In the export screen (https://mongodb.invisionapp.com/share/BVNOZB4DYT3#/screens/347756471) the code snippet with filter, project, sort or limit is not editable
  • We export the result of the query (or the full dataset) and not only what is displayed by Compass in the first page of results


 Comments   
Comment by Githook User [ 30/Oct/19 ]

Author:

{'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com', 'name': 'Lucas Hrabovsky'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 29/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Comment by Githook User [ 23/Oct/19 ]

Author:

{'name': 'Lucas Hrabovsky', 'username': 'imlucas', 'email': 'hrabovsky.lucas@gmail.com'}

Message: COMPASS-3013: update import-export plugin (#1828)

  • COMPASS-3013: update import-export plugin

This PR introduces many performance enhancements, bug fixes, and new features for the Import/Export plugin. It covers many of the stories in COMPASS-3013

    1. Description

> Combined release notes from mongodb-js/compass-import-export#5 and mongodb-js/compass-import-export#6

      1. Switch to proper node.js streams for processing

The backend has been entirely rewritten to use node.js streams. Import and export also now send the `

{explicitlyIgnoreSession: true}

` option to the node driver which has resolved transient errors and allows very large imports and exports that are performant while correct.

![Screenshot 2019-10-15 11 16 49](https://user-images.githubusercontent.com/23074/66952663-164e4900-f02b-11e9-8ea6-085825903ec9.png)

> Fixes COMPASS-3853(https://jira.mongodb.org/browse/COMPASS-3853)

      1. Import
  • Automatically detect if JSON files are `--jsonArray` or ndjson|jsonl. It just works.
  • No more weird caveats around including empty lines at end of your file or not. It just works.
  • Use the CSV header line for field names
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3827(https://jira.mongodb.org/browse/COMPASS-3827) and COMPASS-3829(https://jira.mongodb.org/browse/COMPASS-3829)

      1. Export
  • Compass can export documents to a valid extended JSON file
  • Export respects project, limit, and skip from query bar
  • Show how many documents it managed to export if there is an error or its canceled
  • Progress is displayed in the export modal as bar + number of docs exported

> Fixes COMPASS-3828(https://jira.mongodb.org/browse/COMPASS-3828) and COMPASS-3825(https://jira.mongodb.org/browse/COMPASS-3825)

      1. Import Options

Adds new user options for import:

  • Ignore empty strings
  • Stop on errors
  • Delimiter (`csv` only)

![Screenshot 2019-10-16 15 46 31](https://user-images.githubusercontent.com/23074/66953255-29ade400-f02c-11e9-8281-8b663604bdec.png)

      1. Updated Progress Bar

From mongodb-js/compass-import-export run:

```
npm run storybook
```

![Screenshot 2019-10-14 21 11 31](https://user-images.githubusercontent.com/23074/66952664-16e6df80-f02b-11e9-8792-72195c9cd641.png)

        1. feat: show file after export

![Screenshot 2019-10-01 15 23 25](https://user-images.githubusercontent.com/23074/65994225-d6029e80-e460-11e9-8f0a-5aca6474580b.png)

        1. feat: pretty human import error messages

![Screenshot 2019-10-03 21 22 48](https://user-images.githubusercontent.com/23074/66175227-c1700300-e626-11e9-8343-963d6f4f5924.png)
![Screenshot 2019-10-03 21 25 22](https://user-images.githubusercontent.com/23074/66175228-c1700300-e626-11e9-82ee-a8e47986c87a.png)

      1. Checklist
  • [x] New tests and/or benchmarks are included
  • [x] Documentation is changed or added
    1. Motivation and Context
  • [x] Bugfix
  • [x] New feature
  • [x] Dependency update
  • [ ] Misc
    1. Types of changes
      <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
Generated at Wed Feb 07 22:34:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.