Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-46805

Validate should limit memory use in its second pass

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.7.0, 4.4.7
    • Affects Version/s: None
    • Component/s: Storage
    • Labels:
      None
    • Minor Change
    • ALL
    • v4.4
    • Execution Team 2020-03-23

      The validate command runs in two passes. The first pass matches hashes of index key/value pairs generated by the documents against hashes of index key/value pairs appearing in the indexes by incrementing/decrementing counters depending on the hash. If any of the counters is non-zero in the end, there is corruption and a second pass will compute complete sets of extra/missing keys corresponding to each non-zero counter. With lots of corruption, this may require an amount of memory similar to or somewhat greater than the size of all indexes, possibly exhausting memory resources in the server.

      Instead, the second pass should make an estimate for the expected memory usage based on the number of non-zero counters, and if this estimate exceeds some predetermined value (either a fixed constant such as 100 MB or 1 GB, or a fraction of available memory) it should warn about potential excessive memory usage and limit itself to collecting keys corresponding to some (maybe just 1 or 2) counters. The validation result should clearly indicate that the results are incomplete and that more corruption exists.

            Assignee:
            gregory.noma@mongodb.com Gregory Noma
            Reporter:
            geert.bosch@mongodb.com Geert Bosch
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: