Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-13533

Investigate changes in SERVER-46805: Validate should limit memory use in its second pass

    XMLWordPrintable

    Details

    • Last comment by Customer:
      true
    • Story Points:
      3
    • Sprint:
      ServerDocs2020: Oct20 - Oct27, ServerDocs2020: Oct27 - Nov3, ServerDocs2020: Nov3 - Nov 10, ServerDocs2020: Nov10 - Nov 17

      Description

      Description

      Downstream Change Summary

      There is a new server parameter maxValidateMemoryUsageMB which limits the amount of memory that can be used by the validate command. The default value is 200MB. If the number of reported extra/missing index entries reported by validate is limited by this limit, the output from validate will specify this. If no extra/missing index entries are able to be reported because of the limit, the output will also specify the minimum amount of memory needed to report at least one extra/missing index entry.

      Description of Linked Ticket

      The validate command runs in two passes. The first pass matches hashes of index key/value pairs generated by the documents against hashes of index key/value pairs appearing in the indexes by incrementing/decrementing counters depending on the hash. If any of the counters is non-zero in the end, there is corruption and a second pass will compute complete sets of extra/missing keys corresponding to each non-zero counter. With lots of corruption, this may require an amount of memory similar to or somewhat greater than the size of all indexes, possibly exhausting memory resources in the server.

      Instead, the second pass should make an estimate for the expected memory usage based on the number of non-zero counters, and if this estimate exceeds some predetermined value (either a fixed constant such as 100 MB or 1 GB, or a fraction of available memory) it should warn about potential excessive memory usage and limit itself to collecting keys corresponding to some (maybe just 1 or 2) counters. The validation result should clearly indicate that the results are incomplete and that more corruption exists.

      Scope of changes

      Impact to Other Docs

      MVP (Work and Date)

      Resources (Scope or Design Docs, Invision, etc.)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jason.price Jason Price
              Reporter:
              backlog-server-pm Backlog - Core Eng Program Management Team
              Participants:
              Last commenter:
              Githook User Githook User
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since reply:
                14 weeks ago
                Date of 1st Reply: