[SERVER-69006] Add more tracing to fle basic insert/update/replace for hedged reads Created: 19/Aug/22  Updated: 29/Oct/23  Resolved: 07/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.1.1, 6.0.2, 6.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: Mark Benvenuto Assignee: Mark Benvenuto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.1, v6.0
Sprint: Security 2022-09-05, Security 2022-09-19
Participants:
Linked BF Score: 35

 Description   

The FLE2 basic_insert.js, basic_update.js and basic_replace.js tests are failing with hedged reads.

We should add more tracing to understand the clusterTime that is returned from the server, the clusterTime of the document doing the insert/update/replace and the clusterTime of the transaction that commits the FLE 2 work



 Comments   
Comment by Githook User [ 31/Oct/22 ]

Author:

{'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com', 'username': 'markbenvenuto'}

Message: SERVER-69006 Remove FLE2 suites out of hedged reads passthrough

(cherry picked from commit 9c604c3daf7465628e96c0a4c1876c45d8d58955)
Branch: v6.1
https://github.com/mongodb/mongo/commit/36c9a671027b8c6569351cebd66d3e0fe3c334a8

Comment by Liubov Molchanova [ 26/Oct/22 ]

Requesting backport as the issue reproduced in BFG-1543163 on v6.1

Comment by Githook User [ 12/Sep/22 ]

Author:

{'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com', 'username': 'markbenvenuto'}

Message: SERVER-69006 Remove FLE2 suites out of hedged reads passthrough

(cherry picked from commit 9c604c3daf7465628e96c0a4c1876c45d8d58955)
Branch: v6.0
https://github.com/mongodb/mongo/commit/aa13d918e5b82b6451a900860c9ff2ac1afc3953

Comment by Githook User [ 06/Sep/22 ]

Author:

{'name': 'Mark Benvenuto', 'email': 'mark.benvenuto@mongodb.com', 'username': 'markbenvenuto'}

Message: SERVER-69006 Remove FLE2 suites out of hedged reads passthrough
Branch: master
https://github.com/mongodb/mongo/commit/9c604c3daf7465628e96c0a4c1876c45d8d58955

Comment by Mark Benvenuto [ 25/Aug/22 ]

While there were other issues with FLE2 not returning the right operationTime, the fundamental issue is that there are two connections made from the tests to mongod/s.

Encrypted DB - we insert/update/replace on this connection
Unencrypted DB - we verify things are encrypted on this connection

Steps that cause problem

  1. Start server at operationTime T0
  2. Do CRUD, advance operationTime to T1
  3. On response, we run https://github.com/mongodb/mongo/blob/c397409b96213f43cff9e0366faecff8e88795de/src/mongo/shell/session.js#L466-L475 but this only updates the "operationTime" on the encrypted db to T1. The unencrypted db is a different session and remains at T0.
  4. Do query on unencrypted db at T0 and therefore not see changes for T1 on secondaries, sometimes.
Generated at Thu Feb 08 06:12:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.