Resume token not found when change stream pipeline changes

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL
    • Hide
      1. create a changestream with filter: `[ { "$match": { "$and": [ {"ns.db": "test"}

        , {"ns.coll": {"$in": ["t1", "t2"]}} ] } } ]`

      2. insert into t1
      3. insert into t2
      4. create a changestream with filter: `[ { "$match": { "$and": [ {"ns.db": "test"}

        , {"ns.coll": {"$in": ["t1"]}} ] } } ]` and attempt to resume from the resumetToken created by t2

      5. insert into t1

       

      Show
      create a changestream with filter: `[ { "$match": { "$and": [ {"ns.db": "test"} , {"ns.coll": {"$in": ["t1", "t2"] }} ] } } ]` insert into t1 insert into t2 create a changestream with filter: `[ { "$match": { "$and": [ {"ns.db": "test"} , {"ns.coll": {"$in": ["t1"] }} ] } } ]` and attempt to resume from the resumetToken created by t2 insert into t1  
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi, we opened a deployment-level change stream with pipeline filtering on db and coll fields, i.e. 
      ```

      [ { "$match": { "$and": [

      {"ns.db": "test"}

      , {"ns.coll": {"$in": ["t1", "t2"]}} ] } } ]

      ```

      We then attempt to resume changestream with an updated filter that only contains t2 (say, because we no longer want to watch t1):

      ```

      [ { "$match": { "$and": [

      {"ns.db": "test"}

      , {"ns.coll": {"$in": ["t2"]}} ] } } ]

      ```

      And this results in the following error when trying to restart changestream with the resumeToken. Further decoding this token, and it's pointing to a document in t2.

      > cannot resume stream; the resume token was not found. {_data: "82687EDB22000000012B042C0100296E5A1004C2BD4A64A5A04F3C8CDF814A7DED4E96463C6F7065726174696F6E54797065003C696E736572740046646F63756D656E744B65790046645F69640064687EDB22245CA5A3B5F47107000004"}

      IIUC, the error has to do with the fact that there's a mismatch between the filter that was used to create the changestream and the filter used to resume the changestream. However, it works just fine if the filter adds a table instead of remove. Trying to understand why it's not possible to scan the oplog until the entry in t2 was found. And also what would be the recommended approach to handle a scenario where i want to modify the pipeline. creating a new changestream without a resumeToken or using StartAtOperationTime does not allow us to pick up from the spot that we last processed. 

            Assignee:
            Yuan Fang
            Reporter:
            Joy Gao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: