[SERVER-51245] Adapt resharding's oplog fetcher to use the `Shard` interface for running commands Created: 30/Sep/20  Updated: 29/Oct/23  Resolved: 25/Nov/20

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Task Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: PM-234-M2, PM-234-T-oplog-fetch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-52593 Integrate ReshardingOplogFetcher into... Closed
Problem/Incident
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2020-11-02, Sharding 2020-11-16, Sharding 2020-11-30
Participants:
Linked BF Score: 0

 Description   

Resharding's oplog fetcher is currently using a DBClientBase for runningĀ an aggregation request to fetch oplog entries.

The DBClientBase omits some critical behavior such as:

  • Not updating the logical clock values when receiving a response
  • Does not perform targetting
  • Will not use connections from a pool

Likewise, Shard has some limitations that made it difficult to use for the first milestone:

  • Lack of aggregation support
  • Does not expose a cursor/streaming interface
    Shard also does not support exhaust cursors which is not necessary for a milestone, but would be an unfortunate lost optimization. It's unlikely that the underlying async Fetcher library that Shard uses will ever be able to make use of exhaust cursors.

When doing this ticket, it may be worth considering some unrealized optimizations when writing documents into the oplog. Specifically, batching an entire response into a single WUOW/transaction without requiring the caller making a copy of the response's buffer of BSON objects.



 Comments   
Comment by Githook User [ 27/Nov/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-51245 Use ThreadClient with launchAsync().

Windows reuses threads with std::async() and so the Client won't
necessarily be destructed otherwise.
Branch: master
https://github.com/mongodb/mongo/commit/b8287c2fca88e46285a250c3b585f393418f6eef

Comment by Githook User [ 26/Nov/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-51245 Fix compile on Windows DEBUG.
Branch: master
https://github.com/mongodb/mongo/commit/7e1171785e8efbdbdc661543607872a41cf8a1e2

Comment by Githook User [ 25/Nov/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-51245: Have resharding oplog fetching use a Fetcher.
Branch: master
https://github.com/mongodb/mongo/commit/2d295d7b205dcac66d434929553610dd48ac0815

Comment by Githook User [ 25/Nov/20 ]

Author:

{'name': 'Uladzimir Makouski', 'email': 'uladzimir.makouski@mongodb.com', 'username': 'umakouski'}

Message: Revert "SERVER-51245: Have resharding oplog fetching use a Fetcher."

This reverts commit cda3a52701fe4143b06bd981b98514e69d0a93eb.
Branch: master
https://github.com/mongodb/mongo/commit/7b92716c97e05c09f624513b03bd3eb7663aa537

Comment by Githook User [ 25/Nov/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-51245: Have resharding oplog fetching use a Fetcher.
Branch: master
https://github.com/mongodb/mongo/commit/cda3a52701fe4143b06bd981b98514e69d0a93eb

Generated at Thu Feb 08 05:24:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.