[SERVER-69626] FLE2 Compact - delete from ESC not in a transaction Created: 13/Sep/22  Updated: 29/Mar/23  Resolved: 29/Mar/23

Status: Closed
Project: Core Server
Component/s: Queryable Encryption
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Mark Benvenuto Assignee: Backlog - Security Team
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Initiative
Assigned Teams:
Server Security
Sprint: Security 2022-10-17
Participants:

 Description   

If the ESC table has N rows, the N-1 rows should be deleted with a delete that is not in a MongoDB user transaction. Normally, all of the ESC operations run in SyncTransactionWithRetries::runNoThrow` which means they run in a TXN. Due to the large number of ESC rows that must be deleted this could create a very large txn, the deletes should instead happen with a delete that does not run as part of the transaction.

1. Extend FLEQueryInterfaceImpl with a new delete method
See FLEQueryInterfaceImpl::countDocuments for example of how this can be done with DBDirectClient
2. Change ESC deletes to use this new method for all but the highest ESC record.

Note: ECC deletes remain under the txn. Also, the ESC null document update remains under the TXN.



 Comments   
Comment by Erwin Pe [ 29/Mar/23 ]

This work has been superseded by the v2 compaction algorithm in OST-1 v6, which was implemented in SERVER-72936

Comment by Mark Benvenuto [ 12/Oct/22 ]

After some more consideration, we need to be a little careful such the changes to ESC are sequenced correctly in relation to the EmuBinary.

The new proposed algorithm with ESC non-transactional deletes is below. The algorithm is the same as the originally but the order of operations is sequenced differently. Within ESC and ECC, the order of how the null record and non-null records.

  1. Start TXN T1
  2. Read ESC
  3. Update ESC null document (This must be updated before the non-txn deletes occur)
  4. Commit T1
  5. Non-transactionally delete all other ESC documents
  6. Start TXN T2
  7. Do ECC compaction as-is
  8. Commit T2
Generated at Thu Feb 08 06:13:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.