Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22049

mongoimport with upsert is very slow on 3.x with writeConcern majority

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 3.0.8, 3.2.0
    • Tools
    • None
    • ALL
    • Hide

      All following tests are done with the primer dataset as mentioned in our documentation.


      In 2.6.9, it works as expected.

      2.6.9 mongoimport

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop
      connected to: 127.0.0.1
      2016-01-04T11:23:04.064-0500 dropping: test.restaurants
      2016-01-04T11:23:06.012-0500 check 9 25359
      2016-01-04T11:23:06.013-0500 imported 25359 objects
      

      2.6.9 mongoimport with upsert

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
      connected to: 127.0.0.1
      2016-01-04T11:23:13.559-0500 dropping: test.restaurants
      2016-01-04T11:23:15.452-0500 check 9 25359
      2016-01-04T11:23:15.455-0500 imported 25359 objects
      


      However, in 3.x the upserted import takes a long time to run. Tested on multiple versions including 3.0.8 and 3.2.0.

      3.0.8 mongoimport

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop
      2016-01-04T11:19:22.904-0500	connected to: localhost
      2016-01-04T11:19:22.904-0500	dropping: test.restaurants
      2016-01-04T11:19:24.406-0500	imported 25359 documents
      

      3.0.8 mongoimport with upset

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
      2016-01-04T11:19:29.025-0500	connected to: localhost
      2016-01-04T11:19:29.025-0500	dropping: test.restaurants
      2016-01-04T11:19:32.022-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:35.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:38.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:41.027-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:44.025-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:47.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:50.024-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:53.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:56.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:59.022-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:02.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:05.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:08.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:11.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:14.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:17.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:20.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:23.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:26.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:29.024-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:32.027-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:35.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:38.026-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:39.418-0500	imported 25359 documents
      

      Show
      All following tests are done with the primer dataset as mentioned in our documentation . In 2.6.9, it works as expected. 2.6.9 mongoimport mongoimport --db test --collection restaurants --file primer-dataset.json --drop connected to: 127.0.0.1 2016-01-04T11:23:04.064-0500 dropping: test.restaurants 2016-01-04T11:23:06.012-0500 check 9 25359 2016-01-04T11:23:06.013-0500 imported 25359 objects 2.6.9 mongoimport with upsert mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert connected to: 127.0.0.1 2016-01-04T11:23:13.559-0500 dropping: test.restaurants 2016-01-04T11:23:15.452-0500 check 9 25359 2016-01-04T11:23:15.455-0500 imported 25359 objects However, in 3.x the upserted import takes a long time to run. Tested on multiple versions including 3.0.8 and 3.2.0. 3.0.8 mongoimport mongoimport --db test --collection restaurants --file primer-dataset.json --drop 2016-01-04T11:19:22.904-0500 connected to: localhost 2016-01-04T11:19:22.904-0500 dropping: test.restaurants 2016-01-04T11:19:24.406-0500 imported 25359 documents 3.0.8 mongoimport with upset mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert 2016-01-04T11:19:29.025-0500 connected to: localhost 2016-01-04T11:19:29.025-0500 dropping: test.restaurants 2016-01-04T11:19:32.022-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:35.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:38.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:41.027-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:44.025-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:47.026-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:50.024-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:53.026-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:56.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:59.022-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:02.025-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:05.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:08.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:11.025-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:14.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:17.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:20.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:23.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:26.022-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:29.024-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:32.027-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:35.022-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:38.026-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:39.418-0500 imported 25359 documents

    Description

      mongoimport with the --upsert flag is much slower than expected on 3.x. When running mongoimport with the --upsert flag on an empty collection, performance should be about equal to mongoimport without the flag. This was the case in 2.6.9 but is no longer true in 3.x.

      Attachments

        Activity

          People

            mikeo@mongodb.com Michael O'Brien
            daniel.hatcher@mongodb.com Danny Hatcher (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: