[SERVER-66460] changeStream does not support index Created: 13/May/22  Updated: 27/Oct/23  Resolved: 15/Jun/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: pickup li Assignee: Chris Kelly
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Comments   
Comment by pickup li [ 16/Jun/22 ]

Thanks for repsonse.

I will request on  feedback.mongodb.com 

Comment by Chris Kelly [ 15/Jun/22 ]

Hello,

Thanks for your patience. I was able to confirm this behavior when testing, and found that there is specific reasoning for this not behaving as you are expecting. You will also see this behavior running Mongo.watch() in mongosh.

Change streams do not monitor events in admin, local, and config databases. MongoDB stores all role information in the admin. system.roles collection in the admin database. Additionally, views are inserted into a system collection even if you are using a test db.

This looks like intended behavior at this time, but I can see how you would expect more capability with the change stream functionality. If you'd like, you can share your request at feedback.mongodb.com to get your concerns on the radar. We're starting to direct new feature requests and improvements to that channel and preferring this JIRA project for bug reports specifically.

You may also want to search and post on the MongoDB Developer Community Forums, as it's possible there are others who have guidance on how to satisfy your use-case in the interim.

Regards,
Christopher
 

Comment by pickup li [ 26/May/22 ]

mongodb replica set environment

using docker-compose to start a mongodb replica set 

```
> git clone https://github.com/yowko/docker-compose-mongodb-replica-set-with-auth.git
 
> cd docker-compose-mongodb-replica-set-with-auth

#change image: mongo:4.2.8-bionic to image: mongo:latest

> vim docker-compose.yaml
 #  image: mongo:latest
 
> bash start.sh

```

 

golang to get  changeStream 

```
> git clone https://github.com/abhirockzz/mongodb-changestreams-processor
 
> cd mongodb-changestreams-processor-master
 

#listen to client

#change pipe to {}; and comment out pipeline,

> vi main.go

#//"go.mongodb.org/mongo-driver/bson"

#//matchStage := bson.D{{"$match", bson.D{{"operationType", bson.D{{"$in", bson.A

{"insert", "update", "replace"}

}}}}}}

#//matchStage := bson.D{{"$match", bson.D{{"operationType", "insert"}}}}

#//projectStage := bson.D{
Unknown macro: {"$project", bson.M{"_id": 1, "fullDocument": 1, "ns": 1, "documentKey": 1}}
}

#//pipeline := mongo.Pipeline{matchStage, projectStage}

#cs, err := coll.Watch(ctx, pipeline, opts) -> cs, err := client.Watch(ctx, mongo.Pipeline{}, opts)
 

#connect to mongodb replica set for changeStream
> MONGODB_URI=mongodb://root:pass.123@127.0.0.1:27037 MONGODB_DATABASE=test MONGODB_COLLECTION=students WITH_RESUME=true go run main.go
 
```
 

fill mongodb some data

fill data

```
rs0:PRIMARY> use test
rs0:PRIMARY> db.inventory.insertMany([ { item: "journal", qty: 25, tags: ["blank", "red"], size: { h: 14, w: 21, uom: "cm" }

},

{ item: "mat", qty: 85, tags: ["gray"], size:

{ h: 27.9, w: 35.5, uom: "cm" }

},

{ item: "mousepad", qty: 25, tags: ["gel", "blue"], size:

{ h: 19, w: 22.85, uom: "cm" }

}
])
 
```

 

get change from changeStream

```
> cat change_events
"insert"
"insert"
"insert"
"insert"
```
 

 

test view、index、user、role will no change event

  • create Index with no change event
    ```
    rs0:PRIMARY> use test
    rs0:PRIMARY> db.inventory.createIndex( { qty: 1 } )
    ```
     
  • create view with no change event
    ```
    db.createView("v1","inventory",\\{$project: {"high":"$size.h", "item":1, "qty":2}})
    ```
     
  • create user with no change event
    ```
    use test
    db.createUser( { user: "accountUser", pwd: "pas$Word", roles: [ "readWrite", "dbAdmin" ] }
    )
    ```
     
  • create role with no change event
    ```
    use admin
    db.createRole(
    Unknown macro: {role}
    ,
    Unknown macro: { resource}

    ,

    Unknown macro: { resource}

    ,

    Unknown macro: { resource}

    ],
    roles: [

    { role: "read", db: "admin" }

    ]
    },

    { w: "majority" , wtimeout: 5000 }

    )
    ```

Comment by Chris Kelly [ 24/May/22 ]

Hello,

In order to look into this further, we're going to need some additional information. If possible, it would be helpful if you could supply the following:

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to the ticket:

  • the mongod/mongos logs covering when you attempt the commands you are running
  • the version of mongodb you are using
  • explain whether you are running a single node, replica set, or sharded cluster (remember that change streams on sharded clusters must be opened on mongos)
  • explain in greater detail how you are opening your change stream

Regards,
Christopher

Comment by pickup li [ 18/May/22 ]

I also noticed that user and role are also not supported on changeStream

Comment by pickup li [ 13/May/22 ]

I tested the changeStream of mongodb. The changes of adding view and index to mongodb are not reflected in changeStream, but the corresponding log can be found in oplog

Generated at Thu Feb 08 06:05:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.