[SERVER-68341] Implement enable/disable command for mongoQ in serverless Created: 26/Jul/22  Updated: 12/Dec/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Rishab Joshi (Inactive) Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: change-streams-w/-mongoq, phase3, pm3278-m2, replace-atlas-proxy-w-mongoq
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Initiative
Related
related to SERVER-69068 Further investigate random failures i... Backlog
Assigned Teams:
Query Execution
Participants:

 Description   

PM-2334 implemented enable/disable change streams command for the replica sets. The corresponding ticket was: SERVER-66631

This work for the sharded cluster was punted because there are subtle differences in how mongoS and mongoQ behave. For example, in mongoQ, the above command should dispatch the request to only a subset of shards where the corresponding tenant resides as opposed to mongoS where the request gets sent to all the shards. Since, there can be up to 10k shards (nodes) in the serverless, fanning out requests blindly to all the nodes is not a scalable solution. This ticket is about extending the work done in the SERVER-66631 to run in the mongoQ.

 

Since we will be building this enable/disable command using the POS service, we don't have atomicity when creating change collections on shards, ie. a change collection at shard1 might be made a timestamp t1 and on shard2 at timestamp t2. Whilst in general, this is not a problem for the change stream, it could be a problem for the following rare occurring scenario:

Consider this:

  1. shard1 change collection creating ts is t1
  2. shard2 change collection creating rs is t2
  3.  t1 < t2
  4. other than the change collection seed-entry (own create-collection oplog entry), no other entries were written to the change collection, ie. each of the change collections has just one entry, ie the seed-entry.
  5. the client immediately opened the change stream cursor and the resume token corresponds to t1.
  6. a get-next on shard2 will result in a ChangeStreamHistoryLost error, as the time t2 > provided time t1.
  7. this case is similar to the addShard case with the change collection.

 

Above is not a problem for the oplog because the oplog gets created at a very early stage of the node-bring up, as such the change stream resume token is always greater than the first entry of all the oplog in all the nodes. 

To mitigate above issue, we could potentially enable a change stream in the mongoQ in 2 stages -

  1. the first stage will be to create a change collection in required shards.
  2. And the second stage will be to add a dummy oplog entry to the tenant's change collection in required shards.

With this, we will ensure that the resume token that the client will operate upon will be greater than the initial timestamp of all the change collections.

While this is just one solution, we will explore more options when we work on this ticket.

 

This work will be under the serverless initiative - Change streams with mongoq



 Comments   
Comment by Kyle Suarez [ 14/Apr/23 ]

Had a quick discussion with bernard.gorman@mongodb.com and while it's not quite the same motivation as the rest of the tickets in the mongos process interface project, it's fine to just tack it on to that project. Removing this from director triage.

Notes from our discussion

  • New implementation would forward the command to shards
  • Main work is probably handling testing and test failures – there are some differences in our expectations sharded vs unsharded
Comment by Bernard Gorman [ 13/Apr/23 ]

sebastien.mendez@mongodb.com, sending this to you for now to find an assignee on your team. Per the conversation above, it seems like we can just broadcast for now and refine the targeting later. We may have to deal with a couple of JS test failures as a result, but we can likely just blocklist those.

Comment by Janna Golden [ 10/Apr/23 ]

bernard.gorman@mongodb.com - ah so we will need these commands in the embedded router layer by 8.0 due to the "all clusters sharded in 8.0" plan laid out by the invisible sharding init. In 8.0, the embedded router layer in Serverless will very likely just be the same as the embedded router layer in dedicated clusters - today mongoq technically exists, though it actually just links in almost the entirety of mongos's link graph (the plan was always for mongoq to be built on top of mongos, and perhaps we'll actually re-evaluate whether mongos and mongoq should truly be different processes). In any case, in this ticket I think we can add the commands to the "embedded router layer"'s commands library, and fail the command if it's run without multitenancySupport enbaled - that way we ensure it isn't run in a dedicated setting.

We don't yet have a "tenant shard targeting" protocol, so for now you could probably broadcast the request, including the tenantId sent in the original request. We can always change the targeting behavior once we've implemented "tenant shard targeting".

Comment by Bernard Gorman [ 07/Apr/23 ]

steven.vannelli@mongodb.com, janna.golden@mongodb.com: sure, we can try to schedule this ticket - but I thought we'd have to wait until more work has been completed on the mongoq roadmap before it's feasible to do so. Is my understanding wrong? Is it actually feasible to do this work now?

Comment by Steven Vannelli [ 07/Apr/23 ]

The Server Serverless team would like to have this ticket completed by 8.0 We have some time but figured we'd put it up for discussion for FY24 Q2 if the team has time. No harm in doing it early  

We might be able to place this in PM-3278 if folks agree with that.  

cc janna.golden@mongodb.com 

Generated at Thu Feb 08 06:10:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.