-
Type: Task
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Replication
-
Fully Compatible
-
Repl 2023-08-21, Repl 2023-09-04, Repl 2023-09-18, Repl 2023-10-02
-
13
Hashing for the extra index keys follows a similar approach as the current data consistency check in dbCheck. After the primary completes reverse lookup for a batch, it will pin an OpTime, iterate through that batch range at that OpTime, and hash the index key values together. The primary will then create an oplog entry with the OpTime, hash, and batch range for secondaries to replicate.
Upon receiving the oplog entry, secondaries will iterate through that same batch range at the same OpTime and expect to generate the same hash. If they are not the same, report an inconsistency to the health log. Since we will perform the hashing portion on both primary and secondaries, we will implement this in a hashing class similar to DbCheckHasher.
It is important to note that on the primary, we will only assign an OpTime to the batch and hash the index keys after we have completed the reverse lookup. This is to avoid pinning an old OpTime while we are slowly looking up each document per index key, which will be slow. This may mean that we run reverse lookup on index keys that were inserted and miss index keys that were updated after the user ran dbCheck. Since we have no guarantees on data that was modified after running dbCheck, this is acceptable.
If the user runs dbCheck with skipLookupForExtraKeys: true, we will skip the reverse lookup phase. In this case, the batch boundaries will be determined by the hashing phase.