Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32946

Multiple change streams cause severe performance drop

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.2
    • Component/s: Performance
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Hide
      • Write documents to the collection at 100+/sec speed. They should contain a timestamp.
      • Open 50+ change streams on the same collection with unique $match conditions. Use the timestamp to measure arrival delay.
      • Experience a huge lag.
      Show
      Write documents to the collection at 100+/sec speed. They should contain a timestamp. Open 50+ change streams on the same collection with unique $match conditions. Use the timestamp to measure arrival delay. Experience a huge lag.

      Opening multiple (10+) change stream cursors causes massive delays (up to several minutes) between database writes and notification arrival. A single change stream (or 2-3 of them) does not produce the same issue.

      In a synthetic test, I wrote 100 small documents per second into a database and listen to changes using change streams. I opened 50 change streams and ran it for 100 seconds. The average delay between DB write and change event arrival was 7.1 seconds; the largest delay was 205 seconds (not a typo, over three minutes).

      MongoDB version: 3.6.2
      Test setup #1: MongoDB Atlas M10 (3 replica set)
      Test setup #2: DigitalOcean Ubuntu box + single instance mongo cluster in Docker

      I used a Node.js client, CPU and memory usage was minimal.
      I tried two ways to set up change streams:

      {{let cursor = collection.watch([
        {$match: {"fullDocument.room": roomId}},
      ]);
      cursor.stream().on("data", doc => {...});}}
      

      and

      {{let cursor = collection.aggregate([
        {$changeStream: {}},
        {$match: {"fullDocument.room": roomId}},
      ]);
      cursor.forEach(doc => {...});}}
      

      Both had the same effect.

            Assignee:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Reporter:
            aedm Gabor Gyebnar
            Votes:
            0 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated:
              Resolved: