[SERVER-15284] Insert loop from the shell is slow in MongoDB 2.6 compare to MongoDB 2.4 Created: 17/Sep/14  Updated: 18/Sep/14  Resolved: 18/Sep/14

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sarbamangal Choudhury Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Participants:

 Description   

I was testing the performance of MongoDB 2.6 and insert is taking more time.

people = ["Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy", "Greg", "Steve", "Kristina", "Katie", "Jeff"];
for(var i=0; i<1000000; i++){
   name = people[Math.floor(Math.random()*people.length)];
   user_id = i;
   boolean = [true, false][Math.floor(Math.random()*2)];
   added_at = new Date();
   number = Math.floor(Math.random()*10001);
   db.test_collection.save( {"name":name, "user_id":user_id, "boolean": boolean, "added_at":added_at, "number":number },{writeConcern: {w:0}});
}

This insert 1M records. MongoDB 2.4 took 7 minutes, MongoDB 2.6 took 14 minutes. The test was done on two servers also. It is AWS server with 4 CPU, 15GB Memory and same I/O speed( Server : m3.xlarge) . Also test was done when no one is accessing the server other than the test.



 Comments   
Comment by Sarbamangal Choudhury [ 18/Sep/14 ]

We can't use bulk insert. This is the example. We insert each record and before insert next record if verifies if the record is already on the collection. Our process is like this:

HL7Message processing:

1. Read the message and verify it is duplicate or not.
2. Verify if this message for new Patient or not.
3. Create new message if it is new.
4. Create new patient if it is new.

So we can't use bulk insert what you have described. We are running the same logic in 2.4, but 2.6 is taking more time for the same logic. I like to know how to improve it or fix this on 2.6.

Comment by Ramon Fernandez Marina [ 18/Sep/14 ]

sarbamangal, a better way way to run your test in 2.6 is using the Bulk Write API as follows:

db.test_collection.drop()
 
var bulk = db.test_collection.initializeUnorderedBulkOp();
people = ["Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy", "Greg", "Steve", "Kristina", "Katie", "Jeff"];
for(var i=0; i<1000000; i++){
   name = people[Math.floor(Math.random()*people.length)];
   user_id = i;
   boolean = [true, false][Math.floor(Math.random()*2)];
   added_at = new Date();
   number = Math.floor(Math.random()*10001);
   bulk.insert({"name":name, "user_id":user_id, "boolean": boolean, "added_at":added_at, "number":number});
}
bulk.execute({w:0});

However, as Thomas pointed out above it's much better to test things using a more realistic workload that matches your specific application.

Since this ticket doesn't show a bug in MongoDB I'm going to close it now, as the SERVER project is for reporting bugs or feature suggestions for the MongoDB server and tools. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. If you have further questions about the Bulk Write API I'd recommend you post it on the mongodb-user group to reach a larger audience.

Regards,
Ramón.

Comment by Thomas Rueckstiess [ 17/Sep/14 ]

Hi Sarbamangal,

2.6 introduced some changes in the shell behavior with respect to write commands and write concern. You will find that if you use the new Bulk Write API, the insert performance will be similar to the one from version 2.4 and even return a full overview of how many operations succeeded. A nice writeup about the difference can be found in this blog article.

Simple loops in the javascript shell are not the best way to determine actual throughput of your cluster. I'd recommend a benchmark via your application and driver with realistic workload to compare performance.

Regards,
Thomas

Generated at Thu Feb 08 03:37:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.