[JAVA-3810] Performance bottleneck in bulk insertion Created: 10/Aug/20 Updated: 27/Oct/23 Resolved: 18/Aug/20 |
|
| Status: | Closed |
| Project: | Java Driver |
| Component/s: | None |
| Affects Version/s: | 3.12.6 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Aziz Zitouni | Assignee: | Ross Lawley |
| Resolution: | Gone away | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
-Server Specs Processors: 2 x Intel Xeon E5-2640 2.50GHz -Client Specs Processors: Intel Core i7-4790 CPU @ 3.60GHz (8CPUs). ~3.1GHz Average data transfer rate: 200 MB/s |
||
| Description |
|
Sample code:
The performance of bulk insertion does not improve when increasing the batch size to 5000 or above. The following are the execution times for batch sizes 100, 1000, 5000 and 10000. batch size 100 batch size 1000 batch size 5000 batch size 10000 |
| Comments |
| Comment by Aziz Zitouni [ 12/Aug/20 ] |
|
Hi Ross, Thanks for the suggestions. I will make the necessary changes to the code and create a new topic on the community forums. Regards, Aziz |
| Comment by Ross Lawley [ 11/Aug/20 ] |
|
Hi azitouni@magnitude.com thank you for reaching out. As this sounds like a support issue, I wanted to give you some resources to get this questioned answered more quickly:
Just in case you have already opened a support case and are not receiving sufficient help, please let me know and I can facilitate escalating your issue. Just to note, your benchmark is also including time that it takes to create the batch operations, which you may wish to remove them from the timings. Also, there are limits to the size of bulk operation that can be sent to MongoDB but the driver will automatically handle splitting any batches, so larger documents may see multiple network round trips. You could also use the Command Montioring feature to give insight to timings as well as that can be used to show the network cost of the command. Thank you! Ross |