[SERVER-35558] Drop in performance for the insert command after restarting the MongoDB database Created: 12/Jun/18 Updated: 27/Oct/23 Resolved: 17/Jun/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Nikolay Malitsky | Assignee: | Dmitry Agranat |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||
| Issue Links: |
|
||||
| Participants: | |||||
| Description |
|
Good afternoon, Our team has experienced a substantial drop in performance for the bulk insert command (when inserting 10,000 documents) after restarting the MongoDB database (3.2.20). Based on a brief analysis, there seem to be two issues: (1) The particular collection uses random scattered keys (2) The index tree is only partially loaded in the CPU memory. The first issue can be easily fixed by prepending timestamp to the scattered keys. For resolving the second issue, we are looking for feedback from MongoDB experts. Specifically, there are three questions:
https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.6.10.18.ipynb
Thank you in advance, Nikolay This question has been submitted to http://groups.google.com/group/mongodb-user and http://stackoverflow.com/questions/tagged/mongodb. |
| Comments |
| Comment by Dmitry Agranat [ 17/Jun/18 ] |
|
Hi nmalitsky, Thanks for your report and questions. As we did not find any evidence that the reported issue is related to MongoDB, I will go ahead and close this ticket. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group If this issue is reproducible on a storage system as mentioned here, feel free to reopen this ticket. Thanks, |
| Comment by Nikolay Malitsky [ 15/Jun/18 ] |
|
Hi Dima, Our IT team is analyzing the file system. I expect that that slow Most importantly, your analysis confirmed my previous concern associated 1. Is there any mechanism to push and keep an index tree in the CPU memory? 2. Are the data and index caches managed by one or two different LRU 3. Is prepending the scattered random keys with timestamp (line 7-8 in 4. What btree and cache information is provided in ‘wiredTiger’ entry https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb These questions are important for updating our MongoDB environment and we Best regards, Nikolay On Thu, Jun 14, 2018 at 2:39 PM, Dmitry Agranat (JIRA) <jira@mongodb.org> |
| Comment by Dmitry Agranat [ 14/Jun/18 ] |
|
Hi nmalitsky, If you can reproduce this issue on the system which has more than just a few MB/s read & write throughput capacity, we'll be able to look into this. For a reference, these are our disk and file-system recommendations.
I was not able to find any indication that the reads from disk were related to MongoDB. If this issue is reproducible on a storage system according to our recommendations (and ideally on the latest MongoDB version), please provide the following information:
Thanks, |
| Comment by Nikolay Malitsky [ 14/Jun/18 ] |
|
Hi Dima, Thank you for the analysis. A rate of ~ 2.8 MB/s looks very slow indeed. Would this rate include some Here is the output for the “df –l” command: /dev/sda7 913578428 355946304 511201916 42% / From our side, profiling clearly identified a performance issue (CPU vs the __page_read method that reads a page from the file: https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/bt_read.c Regards, On Thu, Jun 14, 2018 at 5:46 AM, Dmitry Agranat (JIRA) <jira@mongodb.org> |
| Comment by Dmitry Agranat [ 14/Jun/18 ] |
|
Hi nmalitsky, Thank you for uploading the requested information. It was very useful. I suspect the issue is related to the i/o subsystem.
To proceed with this investigation, could you please provide more details about the sda disk and what filesystem type is used? Also, is there any specific reason you are not running with latest MongoDB 3.6.5 version? Thanks, |
| Comment by Nikolay Malitsky [ 13/Jun/18 ] |
|
Hi Dima, Thank you for your prompt response. At this time, our production system is running a fresh database (with a few The performance issue is reproducible. To address your request I ran the 1. Start the MongoDB database (at 10:33 am EDT, June 13, 2018). This malitsky@gpu-001:~/mongo/bench.v3.2$ 2018-06-13T10:32:51.849-0400 I CONTROL [main] log file 2. Run the python script for inserting 10,000 documents (at 10:49 am EDT, https://github.com/malitsky/nsls2/blob/master/mongohxn/tests/insert.gpu-001.6.13.18.ipynb The corresponding diagnostics.data.tar.gz and log.tar.gz files are Based on preliminary profiling, the issue (CPU vs Wall time) is https://github.com/mongodb/mongo/blob/v3.2/src/third_party/wiredtiger/src/btree/row_srch.c Thanks again, Nikolay On Tue, Jun 12, 2018 at 4:39 PM, Dmitry Agranat (JIRA) <jira@mongodb.org> |
| Comment by Dmitry Agranat [ 12/Jun/18 ] |
|
Hi nmalitsky, So we can investigate it can you please upload to this secure private portal, for the affected instance:
Also can you please give me a timeline of the events that you describe as accurately as you can (including timezone). This will ensure that we're looking at the right time period in the data. Thanks, |