[SERVER-32243] Add an option to have the validate hook skip some collections. Created: 08/Dec/17 Updated: 30/Oct/23 Resolved: 08/Jan/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | 3.7.1 |
| Fix Version/s: | 3.4.12, 3.6.3, 3.7.1 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Robert Guo (Inactive) | Assignee: | Jonathan Abrahams |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v3.6, v3.4, v3.2
|
||||||||||||||||
| Sprint: | TIG 2018-1-15 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
For certain workloads that require large oplogs, it may be desirable to not validate the oplog. This ticket intends to have the validate hook ignore collections that are passed in through TestData: e.g. TestData.validateIgnoredNS=['local.oplog.rs']. Collection validation in the hook is done here. In more detail. validate does 3 things: check the document is valid BSON, check indexes are valid, check the WT table structure. On the oplog, validate can only do the BSON check. Because there are no indexes to validate on the oplog and WT can't verify the table on a live replica set, because it can't get exclusive access. Effectively we'd only be missing the BSON validation. Which I think is not as crucial on the oplog as on user collections. The oplog is always read, so we'd be converting documents in it into BSON anyway, the process of which should uncover some invalid BSON issues. Compare this with if a user's document's BSON gets corrupted when written to disk. if the document is never read again, we'd never find out about it, unless we run validate(). |
| Comments |
| Comment by Githook User [ 30/Jan/18 ] | |||
|
Author: {'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}Message: | |||
| Comment by Githook User [ 30/Jan/18 ] | |||
|
Author: {'email': 'jonathan@mongodb.com', 'name': 'Jonathan Abrahams', 'username': 'hptabster'}Message: (cherry picked from commit 56ba266ca7eb46bfca0dc15ba0ca2290237db713) | |||
| Comment by Githook User [ 16/Jan/18 ] | |||
|
Author: {'email': 'henrik.ingo@mongodb.com', 'name': 'Henrik Ingo', 'username': 'henrikingo'}Message: (cherry picked from commit 0784425fa2d58b6a2bff3125b50be7f0d6a7f489) | |||
| Comment by Githook User [ 15/Jan/18 ] | |||
|
Author: {'email': 'henrik.ingo@mongodb.com', 'name': 'Henrik Ingo', 'username': 'henrikingo'}Message: (cherry picked from commit 0784425fa2d58b6a2bff3125b50be7f0d6a7f489) | |||
| Comment by Githook User [ 15/Jan/18 ] | |||
|
Author: {'email': 'henrik.ingo@mongodb.com', 'name': 'Henrik Ingo', 'username': 'henrikingo'}Message: | |||
| Comment by Jonathan Abrahams [ 10/Jan/18 ] | |||
|
henrik.ingo Try using
| |||
| Comment by Robert Guo (Inactive) [ 10/Jan/18 ] | |||
|
henrik.ingo I realized there needs to be a couple of changes. 1. The name of the option was changed to skipValidationNamespaces from validateIgnoredNS. 2. You're right that TestData needs to be defined as well. So you'll need something like this:
| |||
| Comment by Henrik Ingo (Inactive) [ 10/Jan/18 ] | |||
|
Hi robert.guo, jonathan.abrahams I'm trying to use this now from the DSI side, by
The first line will give an error. Did you perhaps mean:
| |||
| Comment by Githook User [ 08/Jan/18 ] | |||
|
Author: {'name': 'Jonathan Abrahams', 'username': 'hptabster', 'email': 'jonathan@mongodb.com'}Message: | |||
| Comment by Robert Guo (Inactive) [ 04/Jan/18 ] | |||
|
Re Kevin: The newly added UUID checks will also be skipped. It's possibly to only validate the UUID by adding another option to the validate() command, but I think the additional complexity and documentation outweigh potential benefits. Since it's unlikely we somehow only fail to generate UUIDs for large collections. Re david.daly The following script should do the job:
I'm a bit hesitant to add this script to the jstests directory since it is used for a very specific purpose; putting it in the perf repo might be better aligned with its use case for the time being. Alternatively, I think there might be a better solution. It should be possible to change this line and line 45 to decouple the name variable from the JS file name. SCRIPT_NAMES can then be made into a dictionary of lambdas that generate bash or mongo shell commands on the fly. I think doing it this way instead of using a different running file will be more flexible and should future-proof needs for JS files without the additional complexity of piping in more configuration options. | |||
| Comment by Kevin Duong [ 12/Dec/17 ] | |||
|
robert.guo How will this impact UUID checking? Seems like there's work needed on the server side? |