[SERVER-49895] Expose getLatestOplogTimestamp() in aggregation cursor command responses on oplog Created: 25/Jul/20  Updated: 29/Oct/23  Resolved: 28/Aug/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: 4.7.0

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Alexander Taskov (Inactive)
Resolution: Fixed Votes: 0
Labels: PM-234-M2, PM-234-T-oplog-fetch, query-work-resharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-49897 Insert no-op entries into oplog buffe... Closed
Related
related to SERVER-51227 Make find/getMore cmd with $_requestR... Closed
related to SERVER-53534 Support including postBatchResumeToke... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2020-09-07
Participants:

 Description   
  • Add syntax to aggregate command to allow QueryPlannerParams::TRACK_LATEST_OPLOG_TS to be set. Some care must be taken to reject user requests which attempt to set the parameter for a non-oplog namespace.
  • Add Timestamp latestOplogTimestamp to the CursorResponse type.
    • CursorResponseBuilder::setLatestOplogTimestamp() should likely be called in all the same places CursorResponseBuilder::setPostBatchResumeToken() is being called today.
    • If latestOplogTimestamp.isNull() then omit the field from the cursor command response.


 Comments   
Comment by Githook User [ 28/Aug/20 ]

Author:

{'name': 'Alex Taskov', 'email': 'alex.taskov@mongodb.com', 'username': 'alextaskov'}

Message: SERVER-49895 Expose getLatestOplogTimestamp() in aggregation cursor command responses on oplog
Branch: master
https://github.com/mongodb/mongo/commit/7d915a65b286e535397a5d79bf2109512003bda9

Comment by Max Hirschhorn [ 27/Jul/20 ]

(or if the query wasn't for the oplog? not sure if that's an expected case, or a scenario of a misbehaving client)

CollectionScan invariants that CollectionScanParams::shouldTrackLatestOplogTimestamp can only be set to true for an oplog namespace. We'll need to turn that into a uassert() as part of this ticket to add user-facing syntax to enable it.

invariant(!_params.shouldTrackLatestOplogTimestamp || collection->ns().isOplog());

Comment by Daniel Gottlieb (Inactive) [ 27/Jul/20 ]

The reason why I said to omit the field from the cursor response when latestOplogTimestamp.isNull() is because the find, aggregate, and getMore commands need not bother to check if they are an oplog cursor. latestOplogTimestamp being a non-null Timestamp should automatically imply that.

That's valuable reasoning, thanks for adding that.

A null timestamp won't ever be recorded in the oplog so I don't see how it could ever be an expected state.

Ok good, glad we're on the same page there.

I don't see a reason a replica set member started as a standalone couldn't make use of this feature.

I didn't mean the second-order interpretation of "standalone", but rather the natural "node that does not have a real oplog". I was trying to understand why internal code querying for the latest oplog timestamp when it doesn't exist shouldn't be considered an error.

My interpretation now is that it's fine for the lower level API to return an error (for example, when the oplog doesn't exist, or when the document found does not include a ts field) – but the expected reason the response building code would observe LatestOplogTimestamp.isNull() is simply "TRACK_LATEST_OPLOG_TS" was not set (or if the query wasn't for the oplog? not sure if that's an expected case, or a scenario of a misbehaving client).

Comment by Max Hirschhorn [ 27/Jul/20 ]

A null timestamp won't ever be recorded in the oplog so I don't see how it could ever be an expected state. CollectionScan::getLatestOplogTimestamp(), PlanExecutor::getLatestOplogTimestamp(), etc. return a Timestamp and not a boost::optional<Timestamp>.

// If _params.shouldTrackLatestOplogTimestamp is set and the collection is the oplog, the latest
// timestamp seen in the collection.  Otherwise, this is a null timestamp.
Timestamp _latestOplogEntryTimestamp;

The reason why I said to omit the field from the cursor response when latestOplogTimestamp.isNull() is because the find, aggregate, and getMore commands need not bother to check if they are an oplog cursor. latestOplogTimestamp being a non-null Timestamp should automatically imply that.

Is that preemptively defensive (e.g: instead of uasserting on standalones)?

I don't see a reason a replica set member started as a standalone couldn't make use of this feature.

Comment by Daniel Gottlieb (Inactive) [ 27/Jul/20 ]

If latestOplogTimestamp.isNull() then omit the field from the cursor command response.

Is that preemptively defensive (e.g: instead of uasserting on standalones)? Or would seeing a null timestamp imply an expected state?

Generated at Thu Feb 08 05:21:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.