Uploaded image for project: 'Compass '
  1. Compass
  2. COMPASS-4547

Export/Import of very large collection loses documents

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Cannot Reproduce
    • Affects Version/s: 1.23.0
    • Fix Version/s: No version
    • Component/s: Import/Export
    • Labels:
      None
    • Environment:
      Compass v1.23.0 on Windows 10
    • Documentation Changes:
      Not Needed

      Description

      Problem Description

      I created a large test collection with 152 million small documents, totalling 12.9GB. I used COMPASS to export it as JSON. The export ran and upon completion, COMPASS displayed a message stating it completed AND also dropped the collection itself (!!). Sadly, I did not grab a screen shot of the message.  No error was displayed, and a collection.json document was created as expected (12.9GB in size).

      I then imported the document into a new collection.  The process ran for slightly longer than the export time... and completed to 100%.  But only 15.1 million documents were created.

      I suspect that it was the EXPORT side that failed, although it reached 100%.. Why it also dropped the source collection is a mystery. I have been unable to reproduce the issue, but will attempt to do so after the many hours spent creating the NOW LOST test data.

      Steps to Reproduce

      I will recreate the large dataset and attempt to reproduce on v1.24.6.

      Expected Results

      Export of large datasets should work fine. 

      Under no circumstances should the Compass Export process also DROP the source collection.

      Import should fail if the import file is syntactically incorrect.

      Actual Results

      Documents were lost, likely during export. Import ran without any complaints. Does this mean the import file has all the closing syntax in place? 

      Additional Notes

      A screen shot of the approximate collection statistics prior to export is attached, along with the statistic after re-importing.

      I can provide the export data file (12.9GB), if requested. It is all dummy data.

       

       

        Attachments

        1. Compass_Export-Import.png
          70 kB
          Lester Waters

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            lesterw@iotahoe.com Lester Waters
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: