[SERVER-22049] mongoimport with upsert is very slow on 3.x with writeConcern majority Created: 04/Jan/16  Updated: 25/Mar/16  Resolved: 25/Mar/16

Status: Closed
Project: Core Server
Component/s: Tools
Affects Version/s: 3.0.8, 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Danny Hatcher (Inactive) Assignee: Michael O'Brien
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Operating System: ALL
Steps To Reproduce:

All following tests are done with the primer dataset as mentioned in our documentation.


In 2.6.9, it works as expected.

2.6.9 mongoimport

mongoimport --db test --collection restaurants --file primer-dataset.json --drop
connected to: 127.0.0.1
2016-01-04T11:23:04.064-0500 dropping: test.restaurants
2016-01-04T11:23:06.012-0500 check 9 25359
2016-01-04T11:23:06.013-0500 imported 25359 objects

2.6.9 mongoimport with upsert

mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
connected to: 127.0.0.1
2016-01-04T11:23:13.559-0500 dropping: test.restaurants
2016-01-04T11:23:15.452-0500 check 9 25359
2016-01-04T11:23:15.455-0500 imported 25359 objects


However, in 3.x the upserted import takes a long time to run. Tested on multiple versions including 3.0.8 and 3.2.0.

3.0.8 mongoimport

mongoimport --db test --collection restaurants --file primer-dataset.json --drop
2016-01-04T11:19:22.904-0500	connected to: localhost
2016-01-04T11:19:22.904-0500	dropping: test.restaurants
2016-01-04T11:19:24.406-0500	imported 25359 documents

3.0.8 mongoimport with upset

mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
2016-01-04T11:19:29.025-0500	connected to: localhost
2016-01-04T11:19:29.025-0500	dropping: test.restaurants
2016-01-04T11:19:32.022-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:35.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:38.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:41.027-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:44.025-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:47.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:50.024-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:53.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:56.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
2016-01-04T11:19:59.022-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:02.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:05.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:08.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:11.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:14.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:17.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:20.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:23.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
2016-01-04T11:20:26.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
2016-01-04T11:20:29.024-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
2016-01-04T11:20:32.027-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
2016-01-04T11:20:35.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
2016-01-04T11:20:38.026-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
2016-01-04T11:20:39.418-0500	imported 25359 documents

Participants:

 Description   

mongoimport with the --upsert flag is much slower than expected on 3.x. When running mongoimport with the --upsert flag on an empty collection, performance should be about equal to mongoimport without the flag. This was the case in 2.6.9 but is no longer true in 3.x.



 Comments   
Comment by Michael O'Brien [ 09/Feb/16 ]

One thing to note is that when using --upsert, mongoimport does not (by design) do writes in parallel, so that it can guarantee consistent ordering of operations. If it did not, then you would get completely different results each time mongoimport --upsert because the parallel updates would race with each other.

Apart from that, the default write concern for mongoimport is majority so lowering it to 1 instead should indeed make it faster.

Generated at Thu Feb 08 03:59:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.