Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-42845

Change streams should return events for user-initiated applyOps commands

    • Query Optimization
    • ALL
    • Query 2019-09-09, Query 2019-09-23, Query 2019-10-07, Query 2019-10-21, Query 2019-11-04

      User-specified applyOps operations can happen in some scenarios we care about, such as using mongomirror to import data into atlas. They may also happen in some backup scenarios and in mongorestore, but I don't know much about that.

      It seems incorrect for change streams to miss events generated in this way, though it's not clear when this might come up. One scenario might be cutting over to atlas and using a resume token from the old cluster.

      From the query perspective, in order to support this, it seems tlike we just have to take out some restrictions such as these filters on lsid and txnNumber to get applyOps to show up.

      As a separate piece of work that is very related, I filed SERVER-44450.

      Original Description

      From jesse, this comment means change streams could miss important operations. I'm not sure why this was deliberately done:

      Change streams deliberately ignore user-initiated applyOps commands, they only generate events from transactions' applyOps oplog entries. (Change streams require a txnNumber and lsid in an applyOps oplog entry.)

      Change streams even ignore the individual oplog entries that are generated when a user-initiated applyOps is executed non-atomically. Such entries have "fromMigrate: true", and change streams filter such entries out.

            backlog-query-optimization Backlog - Query Optimization
            judah.schvimer@mongodb.com Judah Schvimer
            0 Vote for this issue
            12 Start watching this issue