Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-74526

Change stream opened against a secondary node uses a lot of CPU even when there is no write load

XMLWordPrintableJSON

    • Query Execution
    • Fully Compatible
    • ALL
    • v6.3
    • Hide

      Start a 2-node replica set. I'm doing it like so:

      $ ./mongo --nodb
> var rst = new ReplSetTest({nodes: 2}); rst.startSet(); rst.initiate();

      Then connect to the secondary node and start watching a change stream on an arbitrary non-existent collection:

      $ mongosh --port 20001
> db.getMongo().setReadPref("secondary");
> var cursor = db.foo.watch();
> while (cursor.hasNext()) { printjson(cursor.next()); }

      You should be able to see the secondary node's CPU utilization spike. The CPU will become idle again if you stop iterating the change stream cursor.

      Show
      Start a 2-node replica set. I'm doing it like so: $ ./mongo --nodb > var rst = new ReplSetTest({nodes: 2}); rst.startSet(); rst.initiate(); Then connect to the secondary node and start watching a change stream on an arbitrary non-existent collection: $ mongosh --port 20001 > db.getMongo().setReadPref( "secondary" ); > var cursor = db.foo.watch(); > while (cursor.hasNext()) { printjson(cursor.next()); } You should be able to see the secondary node's CPU utilization spike. The CPU will become idle again if you stop iterating the change stream cursor.
    • QE 2023-03-06
    • 20

      The changes from SERVER-69959 introduced a performance regression starting in 6.3 for change streams opened against a replica set secondary node. Let's assume that the system is completely idle, with no reads or writes being issued by clients. The client opens a $changeStream against the idle system. The client's driver will issue getMore operations against this cursor in a loop in order to watch for changes. The expected behavior is that the thread executing the $changeStream will block on the server side waiting for inserts to the oplog. This thread should use very little CPU, since it waits on a condition variable rather than busy waiting.

      The changes from SERVER-69959 appear to have caused the change stream thread to wake up far more often than it should. I added experimental logging to the server which showed that in 6.2, a change stream thread watching a secondary node of an idle system would wake up less than 10 times per second. For the same scenario in 6.3, however, the thread wakes up at least a few orders of magnitude more often. The consequence is that the change stream thread behaves effectively like it is busy waiting rather than blocking and has unreasonably high CPU utilization.

      Again, this happens only for change streams opened against secondary nodes, not for change streams opened against primary nodes. I speculate it has something to do with differing behavior between primary and secondary nodes for how we choose which timestamp to read from?

            Assignee:
            david.storch@mongodb.com David Storch
            Reporter:
            david.storch@mongodb.com David Storch
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: