Summary

What is the problem or use case, what are we trying to achieve?

The idea is to transfer the logic and function of filtering/executing change stream pipelines to application server. Currently supported pipeline stages for change stream are anyway only mutation of change document and change streams do not allow any additional data reading from database. That means that the change document provided to change stream holds maximum amount of information and the information can only be reduced or thrown away. The only way how to gain more information in the change document is (AFAIK) to use options `{ fullDocument: "updateLookup", fullDocumentBeforeChange: "whenAvailable" }` but those can be still called on watch set-up. I believe that implementing the small subset of available stages to drivers is quite simple and does not require some special alghoritms or optimalizations.

Bottom line is, that Meteor.js does something similar (I'm not sure about latest version), when it reads oplog and decides on application server whether the read change should be considered change in watched data. A lot of change streams (at leas in our application) just sits open and receive change only once a while. I understand that for some applications this could cause some perf. issues. That's why this feature should probably be opt-in.

The issue and reason behind this proposal is, that when single application run multiple similar change stream watchers it needs to create multiple connections to a database server. A number of connections is limited resource and biggest problem is that after a limit is reached, there is great drop in performance and each request takes few second to receive answer. Increasing the number of possible connections to a db server is not an option since they would still be all used soon. Hence, it might be significant improvement to somehow join together watchers or transfer the operation of filtering to the application server.

Now, I understand, that reason why change streams are processed by database is mostly security, where you can apply user access rules to change stream requests, but for great part of application something like this is not needed since the application server has complete access to all the collections. Of course this mode of change stream filtering would be subject to appropriate user access rights.

This feature should probably be opt-in or can be automatically decided based on user rights and database configuration. It could be maybe decided between db server and driver based on target of change stream and user rights to this target and if it is possible, then driver and db server would switch to this mode.

Motivation

Who is the affected end user?

Who are the stakeholders?

Not sure who the stakeholders are, but i believe that this would improve overal efficiency of database server, mostly in terms of connections usage, while giving only small overhead to application server. This would be applicable to great amount of use-cases. So end-user of an application would benefit from this, same as owner of the system where he would save on infrastructure costs.

How does this affect the end user?

Are they blocked? Are they annoyed? Are they confused?

Users are overall affected by performance drop when connection limit is nearing.

How likely is it that this problem or use case will occur?

Main path? Edge case?

When application creates multiple change streams for single user for his special/individual data requests, then reaching connections limit can happen quite fast and can be guaranteed it will happen.

If the problem does occur, what are the consequences and how severe are they?

Minor annoyance at a log message? Performance concern? Outage/unavailability? Failover can't complete?

Consequences are significant performance drop for all the users of an application sometimes even resulting in outage or impossibility to use application in normal manner.

Is this issue urgent?

Does this ticket have a required timeline? What is it?
For our system it is quite urgent, we are currently fighting with approximately 700 conections under quite small load. There are no connections leaks and only other option instead of this solution for us is to implement similar functionality at application level which although might be unstable and cumbersome.

Is this ticket required by a downstream team?

Needed by e.g. Atlas, Shell, Compass?
No

Is this ticket only for tests?

Does this ticket have any functional impact, or is it just test improvements?
It has functional, but mostly performance-wise impact.{}

Acceptance Criteria

What specific requirements must be met to consider the design phase complete?

• Consider all the relationships of change streams and which of them are really dependant od db server, resp. needs to happen on db server side.
• Consider opt-in/opt-out/automatic decision to use this feature.
• Consider changes needed in all the drivers.

I would like to propose to test this feature out on node.js driver.