[SERVER-85448] Mongodb write performance issue in v7.0 Timeseries Collection Created: 17/Jan/24  Updated: 30/Jan/24

Status: Needs Verification
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: santhosha MS Assignee: Server Triage Escalation
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Mongodb 7.0 version


Attachments: PNG File image (10).png     PNG File image (8).png     PNG File image (9).png    
Assigned Teams:
Server Triage
Participants:

 Description   

we are finding slowness/inconsistency in write speed on mongodb 7.0 timeseries collection.
On 7.0.4 installation with 6.0 feature set and we are able to injest 250mn events an hour (batch size 100 events) with avg latency of 100microsecs per event. Once we switch to 7.0.4 feature set, the avg latency turns out to 500 microseconds per event and eventually increases more than 1000 microseconds. Sometimes it increases and decreases and shows inconsistency in write speed. After writing few milion events, if we switch again to 6.0 feature set, the write avg latency switch back to 100 microseconds per event and it stays consistent even after billions of events injestion. We observed the same with 6.0 installation too not just with 7.0.4 installation with 6.0 feature set.

Create collection query:
db.createCollection("weather",{timeseries:{timeField:"time", metaField:"meta",granularity:"hours"}})

we do insert many query

mongo is installed with 7.0.4 version
command used for switching feature set:
db.adminCommand( { setFeatureCompatibilityVersion: "7.0", confirm: true } )
db.adminCommand( { setFeatureCompatibilityVersion: "6.0", confirm: true } )

command used for checking feature set:
db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )



 Comments   
Comment by Dhanasekaran S [ 30/Jan/24 ]

Hi jessica.sigafoos@mongodb.com ,

Any update on this ticket. I am seeing it is under verification.  If your team needs more info or simulation of this issue, we are open to provide necessary support.

Thanks,

Dhanasekaran

Comment by Dhanasekaran S [ 23/Jan/24 ]

Please find below metrics captured on 6.0 and 7.0 feature sets.  At 23:00 we switched from 6.0 to 7.0.
Please consider these if they can trigger some clues to find the RCA.

 

1) Execution Time for Writes

We observe inconsistency in writes on 7.0 (after 23:00)

 

2) Updates

We are not doing any updates explicitly.  All are inserts into timeseries collection.  But we are seeing updates underhood.  It is clearly seen that in 7.0, the updates per seconds lowered down drastically.

 

3) Query metrics

The Queries per second is way higher in 7.0 compared to 6.0

Comment by santhosha MS [ 18/Jan/24 ]

<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongodb-driver</artifactId>
    <version>3.12.14</version>
</dependency>
<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongodb-driver-core</artifactId>
    <version>3.12.14</version>
</dependency>
<dependency>
    <groupId>org.mongodb</groupId>
    <artifactId>mongo-java-driver</artifactId>
    <version>3.12.14</version>
</dependency>

These are the dependency being used in java code which is doing the insertion.
we are using this below function for inserting data.
timeSeriesCollection.insertMany(docArray);

Comment by Rhys Howell [ 18/Jan/24 ]

santhoshms511@gmail.com Do you know if this is a server issue and not a mongosh issue? Does this same performance impact happen when using a MongoDB driver directly? Thanks for opening the issue, hopefully we can get to the cause of it.

Comment by PM Bot [ 17/Jan/24 ]

Hello santhoshms511@gmail.com, thank you for reaching out to us! The team will review your issue and get back to you soon as soon as possible.

Please review your issue to ensure you've included your environment details and have attached relevant logs (with any sensitive data redacted), so that we're best able to provide you a timely and thorough response. Thanks again!

Generated at Thu Feb 08 06:57:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.