ValidateTests can fail sporadically due to KeyString hashing

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 4.4.7, 5.0.0-rc0
    • Affects Version/s: None
    • Component/s: Storage
    • Fully Compatible
    • ALL
    • v4.4
    • Execution Team 2021-05-03, Execution Team 2021-05-17, Execution Team 2021-05-31
    • 18
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      As witnessed in a build failure that tried to create corruption, a record was removed with {_id: 1} and another record with {_id: 3} was inserted in its place to create a mismatch between the record store and index entries in order to have the validate command detect the corruption.

      Validation didn't detect any inconsistencies in this particular scenario because of the following:

      • The record with {_id: 3} has the KeyString value "2B06040018" and gets hashed to "39020".
      • The index entry for the removed record (_id: 1}, has the KeyString value "2B02040008" which also gets hashed to "39020".

      The IndexConsistency class only has 65536 buckets so it is possible for a collision to occur once in a while.

       

      "validating collection","attr":{"namespace":"unittests.validate_tests","uuid":" (UUID: 57683f78-4573-4aa8-a1ed-a36c725f783f)"}
       ...
      "[validate]","attr":{"recordId":{"RecordId":3},"recordData":{"_id":3}}
      "[validate](record) {hash_num}","attr":{"hash_num":39020}
      "{caller} {record_id}, key: {rehydrated_key}, keystring: {key_string}","attr":{"caller":"[validate](record)","record_id":{"RecordId":3},"rehydrated_key":{"_id":3},"key_string":"2B06040018"}
       ...
      "validating index consistency","attr":{"index":"_id_","namespace":"unittests.validate_tests"}
      "[validate](index) {hash_num}","attr":{"hash_num":39020}
      "{caller} {record_id}, key: {rehydrated_key}, keystring: {key_string}","attr":{"caller":"[validate](index)","record_id":{"RecordId":1},"rehydrated_key":{"_id":1},"key_string":"2B02040008"}
      ... 
      "Validation complete for collection. No corruption found.","attr":{"namespace":"unittests.validate_tests","uuid":" (UUID: 57683f78-4573-4aa8-a1ed-a36c725f783f)"}
      

            Assignee:
            Yuhong Zhang
            Reporter:
            Gregory Wlodarek
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: