[SERVER-37800] ChangeStream performance issues Created: 29/Oct/18  Updated: 26/Nov/18  Resolved: 29/Oct/18

Status: Closed
Project: Core Server
Component/s: Performance
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Ihar Knyshau [X] Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File ChangeStreamProfiling_1.PNG     PNG File ChangeStreamProfiling_2.PNG    
Issue Links:
Duplicate
Participants:

 Description   

For last 3 monthes our team has issues with MongoDB ChangeStream performance.

 

Numbers that we want to achieve - stable 15k documents/sec for 3 ChangeStream consumers over independent collections/databases. 

Current performance numbers - about 7-8k for 1 ChangeStream consumer.

 

Question: How can we achieve desired performance numbers on current cluster setup?

 

MongoDB Cluster deployed on AWS and has following topology:

  Shards: 6 shards with PSA arcitecture. Primary/Secondary - m5.xlarge nodes with EBS disks (2TB of disk space and 6k IOPS for each shard), arbiter - t2.medium node

  Mongos: 3 m5.xlarge nodes, write load balanced across all of it

  Config server: replica set of 3 m5.large nodes

 

Key points of our work:

  • changing number of shards and proper collection sharding don't boost performance
  • increasing instance type for shards and disk shape has low impact on performance
  • monitoring for cluster don't show any kind of bottlenecks. CPU, memory, disk usage, networking values for shards and mongos are in norm
  • increasing document size degrade ChangeStream performance. We pass _id field only
  • increasing number of writes decrease ChangeStream performance: 8k+/sec writes - 8k/sec consume, 12k/sec writes - 5k/sec consume, 14k/sec writes - 3k/sec consume
  • tuning MongoDB server side settings don't improve performance: decreasing waitForSecondaryBeforeNoopWriteMS, increasing of __ ShardingTaskExecutorPoolMaxConnecting and __ replWriterThreadCount
  • tuning client side settings don't improve performance: changing of batch size, number of connections, filter for inserts and updates
  • profiling of client side show large waiting time on acquiring of ChangeStream notification (detailed info in attachments)

 

 

 



 Comments   
Comment by Danny Hatcher (Inactive) [ 29/Oct/18 ]

Thanks for your report. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Generated at Thu Feb 08 04:47:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.