[SERVER-48431]  MongoDB concurrent queries on different collections are slow Created: 27/May/20  Updated: 27/Oct/23  Resolved: 30/Jun/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Concurrency, Performance
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Ajinkya Surnis Assignee: Dmitry Agranat
Resolution: Community Answered Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongo_logs.tar     Text File perf_test3.py     PNG File swaping.png    
Participants:

 Description   

There is linearly increasing latency observed with multiple concurrent aggregations on different collections.

Scenario:

Multiple mongodb collections are created with 100,000 documents in each collection and 10,000 columns in each document. There is a python script which executes aggregate queries in a multi-threaded fashion. Each thread invokes an aggregate on a separate collection. (The code of script is attached with this ticket).

When the script is executed, the amount of time it takes to complete the aggregation is proportional to the number of threads. i.e. the latency linearly increases with number of queries concurrently executed. If single aggregation query takes 'x' amount of time, running 'n' queries concurrently takes almost 'n*x' amount of time.

The expectation was that the multi-threaded queries would take roughly the same amount of time as the single-threaded one.  Is this the known limitation of mongodb? Is there any configuration parameter in mongodb to control concurrency?



 Comments   
Comment by Dmitry Agranat [ 30/Jun/20 ]

Hi surnisaa@gmail.com,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 07/Jun/20 ]

Hi surnisaa@gmail.com, thank you for providing the requested information.

Since you have not mentioned the time frame for this test, I assumed it was executed between 2020-06-02T05:31:07.205Z and 2020-06-02T05:34:20.715Z. If this is not the case, please let us know by posting the exact time frame and time zone of this test.

During the test in question, you've hit the HW bottleneck (memory), CPU (kernel).

I recommend adjusting your HW (adding more than 1 GB RAM and 1 Core) to see if this improves the results.

Thanks,
Dima

Comment by Ajinkya Surnis [ 02/Jun/20 ]

Attaching the diagnostic data and mongo demon logs tar file. mongo_logs.tar

Comment by Carl Champain (Inactive) [ 28/May/20 ]

Hi surnisaa@gmail.com,

Thank you for the report.
Would you please archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) and attach it to this ticket?

Kind regards,
Carl

Generated at Thu Feb 08 05:17:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.