Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-42845

Change streams should return events for user-initiated applyOps commands

    XMLWordPrintable

    Details

    • Operating System:
      ALL
    • Sprint:
      Query 2019-09-09, Query 2019-09-23, Query 2019-10-07, Query 2019-10-21, Query 2019-11-04

      Description

      User-specified applyOps operations can happen in some scenarios we care about, such as using mongomirror to import data into atlas. They may also happen in some backup scenarios and in mongorestore, but I don't know much about that.

      It seems incorrect for change streams to miss events generated in this way, though it's not clear when this might come up. One scenario might be cutting over to atlas and using a resume token from the old cluster.

      From the query perspective, in order to support this, it seems tlike we just have to take out some restrictions such as these filters on lsid and txnNumber to get applyOps to show up.

      As a separate piece of work that is very related, I filed SERVER-44450.

      Original Description

      From A. Jesse Jiryu Davis, this comment means change streams could miss important operations. I'm not sure why this was deliberately done:

      Change streams deliberately ignore user-initiated applyOps commands, they only generate events from transactions' applyOps oplog entries. (Change streams require a txnNumber and lsid in an applyOps oplog entry.)

      Change streams even ignore the individual oplog entries that are generated when a user-initiated applyOps is executed non-atomically. Such entries have "fromMigrate: true", and change streams filter such entries out.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              backlog-query-optimization Backlog - Query Optimization
              Reporter:
              judah.schvimer Judah Schvimer
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: