[SERVER-36835] High mongod CPU usage on multiple change streams Created: 24/Aug/18 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Performance |
| Affects Version/s: | 3.6.6 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Dmitry Volodin | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | change-streams-improvements | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
MongoDB shell version v3.6.6 Pymongo 3.7.1 |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Execution
|
||||||||
| Participants: | |||||||||
| Description |
|
We've noticed increased mongod CPU usage when trying to watch change streams. One of our microservices tracks changes on 6 specially prepared collections and notifies clients when change occures. Service implemented in python (Pymongo 3.7.1). Part of relevant code below
Each service starts separate thread per each change stream (6 in total), we have total of 3 services in our cluster. Tracked collections are relatively rarely changed (100-1000 changes per day). Setup works as expected on single-node replica set. When trying to deploy on 3-node replica set we observing additional significant CPU consuming by mongod service. Our service remains fully functional and works as expected. mongod CPU usage chart is attached. mongod also reports slow getMore operations on tracked collections
Altering pipeline options (including empty pipelines) leads to no visible effect. We've replaced our waiter thread with simple poller
and observed CPU usage dropped to usual levels. Is this expected behaviour? How can we estimate change streams watching penalty? |
| Comments |
| Comment by Joe Kanaan [ 09/Mar/22 ] | ||||||||||||||||||||||||
|
3.6 is EOL | ||||||||||||||||||||||||
| Comment by Aleksey Shirokih [ 28/Aug/18 ] | ||||||||||||||||||||||||
|
collStats files on the store | ||||||||||||||||||||||||
| Comment by Dmitry Volodin [ 28/Aug/18 ] | ||||||||||||||||||||||||
|
We have a set of collections, called DataStreams. Each DataStream contains specially prepared set of data which may be tracked by external systems. DataStream API supports long polling mode, holding incoming request until appropriative collection been changed.
This scheme works as expected for single-node mongo, but implies significant penalty on 3-node cluster. DataStream API is below https://docs.getnoc.com/microservices/en/api/datastream.html | ||||||||||||||||||||||||
| Comment by Nick Brewer [ 27/Aug/18 ] | ||||||||||||||||||||||||
|
Volodin, freeseacher I'd like to confirm a few things:
Thanks, | ||||||||||||||||||||||||
| Comment by Aleksey Shirokih [ 24/Aug/18 ] | ||||||||||||||||||||||||
|
Hi. i will assist Dmitriy with owr case. files uploaded ok | ||||||||||||||||||||||||
| Comment by Nick Brewer [ 24/Aug/18 ] | ||||||||||||||||||||||||
|
Volodin Thanks for your report. Could you please upload:
I've created a secure portal where you can upload this information; it will only be visible to MongoDB employees, and is automatically removed after a period of time. Thanks, |