[SERVER-46414] add change stream notifications for more DDL operations Created: 26/Feb/20  Updated: 29/Oct/23  Resolved: 12/Apr/22

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Change streams
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: Improvement Priority: Minor - P4
Reporter: vinllen chen Assignee: Backlog - Query Execution
Resolution: Fixed Votes: 1
Labels: change-streams-improvements, pm1950-m4, pull-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-52254 Enable feature flag for Improved chan... Closed
Related
is related to SERVER-42845 Change streams should return events f... Closed
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Participants:
Case:

 Description   
Original Summary

mongodb create/delete index should trigger change stream

Original Description

I've already posted a question on [stack overflow|https://stackoverflow.com/questions/60368424/mongodb-create-delete-index-wont-trigger-change-stream] but no response.

Why change stream can't receive DDL events like create/delete index, applyOps, convertToCapped, and so on? I think it's important for users that using change-stream to receive all the change events from MongoDB, not only DML but also DDL.



 Comments   
Comment by Bernard Gorman [ 12/Apr/22 ]

Hi cvinllen@gmail.com,

As of MongoDB 6.0, change streams will report the following DDL events when the new showExpandedEvents parameter is set to true:

  • create (when a collection is created)
  • modify (when a property of a collection or index is modified)
  • createIndexes
  • dropIndexes
  • shardCollection

Please consult our forthcoming 6.0 documentation for full details.

Best regards,
Bernard.

Comment by phoenix Liu [ 19/Feb/21 ]

Hi bernard.gorman,

Thanks for the reply. And very glad to hear that you are handling this ticket and also panning to expand the change streams framework. 
Admit it or not, the original design of change streams has some problems. Otherwise you will not support '$replaceRoot' and then encounter problems like https://jira.mongodb.org/browse/SERVER-37786  

Looking forward to to seeing new solutions and architectures about change streams which has been proved to be a great feature.

BTW, I totally understand your rules about PRs and master branch, it's reasonable. And I think this PR is more like a 'For some reasons I'm using 4.0 version right now, so i have to do this enhancement based on this version' thing. And solution in this PR is NOT perfect based on current change streams framework, as i mentioned before, shard collections is not handled gracefully.

Be free to close the PR. Looking forward to your progress and new version of MongoDB!

Best regards,

Phoenix

Comment by Bernard Gorman [ 16/Feb/21 ]

Hi phoenixxliu@tencent.com,

Many thanks for submitting this pull request! Unfortunately, I'm afraid we cannot accept it at present, for a number of reasons:

(1) We are currently in the early stages of a longer-term initiative to expand and enhance the change streams framework. Exposing additional events is definitely on our roadmap, but we plan to rework the foundations of the system first, and we do not wish to complicate this process - which is already in progress - by adding new features at this stage.

(2) Before we commit to exposing these additional events, we will need to go through an internal process of deciding what operations we wish to make visible, whether we want to expose these events unconditionally or add a new parameter allowing users to choose whether to see them, and - perhaps most importantly - reach consensus on what the exact format of the new events will be. Since we wish to keep change streams' external format stable across releases, it is important to take the time to get these decisions right before moving to implementation.

(3) This PR is based on our 4.0 branch. With the exception of fixing bugs which only affect specific versions, we never make changes to older branches directly; this would result in some functionality being available on earlier versions which are not present in later releases. We always develop new features on our master branch, and then - where feasible and appropriate - backport to earlier branches.

As I mentioned above, adding DDL events to change streams is something we intend to consider in the future, so please do keep an eye on this ticket for further developments!

Best regards,
Bernard

Comment by phoenix Liu [ 18/Jan/21 ]

Hey guys,

I have created a [PR on v4.0|https://github.com/mongodb/mongo/pull/1392] which try to add change stream notifications for more DDL operations like create, createIndexes, dropIndexes, collMod, etc. Maybe someone in appropriate team could take a moment to review it.

(PS: need more considerations and discussion about shard collections)

Comment by Carl Champain (Inactive) [ 28/Feb/20 ]

Hi cvinllen@gmail.com,

Thank you for the report. We're assigning this ticket to the appropriate team for further investigation. Updates will be posted on this ticket as they happen.

Kind regards,
Carl
 

Generated at Thu Feb 08 05:11:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.