[SERVER-43086] mongod does not always include operationTime and clusterTime in responses Created: 29/Aug/19  Updated: 29/Mar/21  Resolved: 28/Oct/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kevin Rosendahl Assignee: Benjamin Caimano (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File reproduce.sh    
Issue Links:
Problem/Incident
Operating System: ALL
Steps To Reproduce:

I've attached reproduce.sh which should show the behavior. It spins up a replica set with --keyFile but no --shardsvr and most of the time operationTime and clusterTime will be present in the first request made, but if not will appear in the second or third.

It then spins up a replica set with --keyFile and --shardsvr, and operationTime and clusterTime will not be present in the response.

Then it spins up a config replica set and a mongos, and adds the replica set as a shard, at which point operationTime and clusterTime will appear in the response.

Participants:

 Description   

When a replica set is started with both --keyFile and --shardsvr, but is not yet added as a shard to a sharded cluster, the responses from the mongod do not contain either operationTime or clusterTime. Once the replica set is added as a shard, the fields begin being populated.

For example, prior to being added as a shard, a response to db.runCommand({ find: 'repro', filter: {} }) looks like (note these are run directly against the replica set):

 

{
        "cursor" : {
                "id" : NumberLong(0),
                "ns" : "test.repro",
                "firstBatch" : [ ]
        },
        "ok" : 1
}

And once it's been added as a shard, it looks like

 

{
        "cursor" : {
                "id" : NumberLong(0),
                "ns" : "test.repro",
                "firstBatch" : [ ]
        },
        "ok" : 1,
        "operationTime" : Timestamp(1567098799, 3),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000001")
        },
        "lastCommittedOpTime" : Timestamp(1567098799, 3),
        "$configServerState" : {
                "opTime" : {
                        "ts" : Timestamp(1567098799, 6),
                        "t" : NumberLong(1)
                }
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1567098799, 6),
                "signature" : {
                        "hash" : BinData(0,"keF7J9EKgnmdyDc/6k4/R2wFh50="),
                        "keyId" : NumberLong("6730638087010910237")
                }
        }
}

 

Additionally, after startup of a replica set with --keyFile but without --shardsvr, it appears there is also a short amount of time before which operationTIme and clusterTime are contained in responses.

I've reproduced this on both 4.0.12 and 4.2.0. I've been unable to reproduce either case without --keyFile.

These fields not being present can have downstream effects on clients which are using a ClientSession and rely on the operationTime or clusterTime being available in that ClientSession after issuing commands.



 Comments   
Comment by Benjamin Caimano (Inactive) [ 24/Oct/19 ]

kevin.rosendahl, I believe this happens not necessarily because of --keyfile but because of authentication in general. If a user doesn't have the undocumented advanceClusterTime privilege action and the replica set doesn't yet have any keys to sign the cluster time, then they will not receive operationTime or $clusterTime. This isn't parsimonious---if you don't have the ability to sign cluster times either on the client or via a mongos, then you cannot meaningfully participate in causal consistency. We can't make this pattern work for you without a major overhaul of how we do key storage for our logical time.

You could theoretically add the advanceClusterTime action to a user role, but please be sure that the client is secure and fully under your control to prevent malicious $clusterTime gossiping.

CC misha.tyulenev as a subject expert.

Comment by Benjamin Caimano (Inactive) [ 23/Oct/19 ]

I've confirmed that this reproduces on master. Investigating further.

Comment by Danny Hatcher (Inactive) [ 30/Aug/19 ]

Because this only reproduces when using the --keyFile setting, giving to the Security team.

Generated at Thu Feb 08 05:02:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.