[DOCS-13533] Investigate changes in SERVER-46805: Validate should limit memory use in its second pass Created: 19/Mar/20  Updated: 13/Nov/23  Resolved: 28/Sep/22

Status: Closed
Project: Documentation
Component/s: manual, Server
Affects Version/s: None
Fix Version/s: 5.0.0, 6.0.0, 6.1.0, Server_Docs_20231030, Server_Docs_20231106, Server_Docs_20231105, Server_Docs_20231113

Type: Task Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Jason Price
Resolution: Done Votes: 0
Labels: docs-administration
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
backported by DOCS-14537 [BACKPORT] [v4.4] Validate should lim... Closed
Documented
documents SERVER-46805 Validate should limit memory use in i... Closed
Participants:
Days since reply: 1 year, 8 weeks, 2 days ago
Epic Link: DOCSP-9747
Story Points: 3

 Description   

Description

Downstream Change Summary

There is a new server parameter maxValidateMemoryUsageMB which limits the amount of memory that can be used by the validate command. The default value is 200MB. If the number of reported extra/missing index entries reported by validate is limited by this limit, the output from validate will specify this. If no extra/missing index entries are able to be reported because of the limit, the output will also specify the minimum amount of memory needed to report at least one extra/missing index entry.

Description of Linked Ticket

The validate command runs in two passes. The first pass matches hashes of index key/value pairs generated by the documents against hashes of index key/value pairs appearing in the indexes by incrementing/decrementing counters depending on the hash. If any of the counters is non-zero in the end, there is corruption and a second pass will compute complete sets of extra/missing keys corresponding to each non-zero counter. With lots of corruption, this may require an amount of memory similar to or somewhat greater than the size of all indexes, possibly exhausting memory resources in the server.

Instead, the second pass should make an estimate for the expected memory usage based on the number of non-zero counters, and if this estimate exceeds some predetermined value (either a fixed constant such as 100 MB or 1 GB, or a fraction of available memory) it should warn about potential excessive memory usage and limit itself to collecting keys corresponding to some (maybe just 1 or 2) counters. The validation result should clearly indicate that the results are incomplete and that more corruption exists.

Scope of changes

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)



 Comments   
Comment by Githook User [ 12/Dec/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533-maxValidateMemoryUsageMB-update (#1902)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: indexes-refactor
https://github.com/10gen/docs-mongodb-internal/commit/9fa823ec1f0d192e8d65b565c0bca2701c4b1450

Comment by Githook User [ 28/Sep/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533-maxValidateMemoryUsageMB-update (#1902) (#1912)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: v5.0
https://github.com/10gen/docs-mongodb-internal/commit/709c9aeb924ff13e1017866996dc94dee1c4f9f8

Comment by Githook User [ 28/Sep/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533-maxValidateMemoryUsageMB-update (#1902) (#1911)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: v6.0
https://github.com/10gen/docs-mongodb-internal/commit/aa0aa56f970e1593b7ee75d806376cce9cca0fca

Comment by Githook User [ 28/Sep/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533-maxValidateMemoryUsageMB-update (#1902)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: master
https://github.com/10gen/docs-mongodb-internal/commit/9fa823ec1f0d192e8d65b565c0bca2701c4b1450

Comment by Vincent Do [ 18/Sep/22 ]

Our docs currently say that this behavior only exists in 5.0+ which is inaccurate. It should be fixed to reflect whatever 4.4 version this was backported to https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.maxValidateMemoryUsageMB

Comment by Githook User [ 16/Jul/21 ]

Author:

{'name': 'Jason Price', 'email': 'jason.price@mongodb.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533 Validate should limit memory use
Branch: v4.4
https://github.com/mongodb/docs/commit/c4fd5ab92d17a8d9c585c115597247228b8f5afc

Comment by Githook User [ 13/Jul/21 ]

Author:

{'name': 'Jason Price', 'email': 'jason.price@mongodb.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533 Validate should limit memory use
Branch: v4.4.7
https://github.com/mongodb/docs/commit/b96b595a6d7f53cf31c69d18553e96c3fc6ee6f1

Comment by Githook User [ 11/Nov/20 ]

Author:

{'name': 'Jason Price', 'email': 'jason.price@mongodb.com', 'username': 'jason-price-mongodb'}

Message: DOCS-13533 Validate should limit memory use
Branch: v5.0
https://github.com/mongodb/docs/commit/458fcce49a4911e975a82fbc55428f8d3e615094

Comment by Ravind Kumar (Inactive) [ 19/Mar/20 ]

Note: There is not a backport ticket associated here at this time. Double check before committing to the 4.4 branch.

Generated at Thu Feb 08 08:08:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.