[SERVER-31401] Document validator fails to reject invalid values in array member Created: 05/Oct/17 Updated: 27/Oct/23 Resolved: 09/Oct/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin, Querying |
| Affects Version/s: | 3.4.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Daniel Smedegaard Buus [X] | Assignee: | Mark Agarunov |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | MAREF | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Steps To Reproduce: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Hi I was trying to add document validation to avoid an issue with our software where a document, containing an array of DBRefs was saved back with those DBRefs resolved to their foreign documents. In other words, to validate that the items in this array member are indeed DBRefs. As DBRef is not available as a $type to check against, I had to get creative and instead opted for this not-exactly-foolprof idea: Either, the member must not exist, or it must be an array of objects which does not have an _id property. In my testing, it becomes clear that it stops rejecting invalid updates once the array contains one valid member. In essence, the validator seems to say, "yep, everything okay" so long as the validator rule, if used as a query, would yield the document being validated. I don't see this behavior mentioned anywhere in the docs, so I'm thinking it's a bug. Please see my steps to reproduce for clarification. |
| Comments |
| Comment by Kelsey Schubert [ 11/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks for your interest in improving our documentation. You can open a DOCS ticket directly using JIRA or by providing feedback on the lower right corner of any page in our documentation. Kind regards, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Smedegaard Buus [X] [ 10/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey guys, thanks for the replies Good to hear about JSON Schema in v3.6! Here's fingers crossed that it'll be an upgrade option with ObjectRocket in the nearish future Is there a JIRA or similar for documentation proposals I can open a ticket in? I still feel like this should be documented. Thanks again! | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 09/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Daniel Smedegaard Buus, Thank you for the response. I hadn't considered empty arrays when testing this, and it does get a bit complex with additional conditions, but I am glad to hear this solved the issue for you. As David Storch mentioned, this should be addressed by JSON Schema in the upcoming MongoDB 3.6 release, so I've closed this ticket as "Works as Designed". Thanks, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 09/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Note that the upcoming MongoDB 3.6 stable release series will add support for JSON Schema. See | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Smedegaard Buus [X] [ 09/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Actually, trying to get this change in, I just found a problem with the approach. You're not allowed to have an empty array, meaning you cannot trust that you can $pull from a member without failing validation (in case that leaves the array empty) :/ I guess one could further modify the rule to accept either the not-not-A clause or a .length of zero... Starts to get really nasty, though | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Smedegaard Buus [X] [ 09/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, Mark! Thanks for that. That's some seriously lateral thinking you did there — respect! I'm super happy that it's possibly to do this in Mongo, and I definitely think that it shouldn't be in the bugs section anymore. I would like to get your thoughts on if perhaps there's room for a behavioural change here, or at least some clear documentation mentions of this? I don't feel like I'll be the first one to not get this, and the problem is that if you don't get it, the result will not be a failing validation rule, it will be admitting unvalidated data into your array members. Since the behaviour of $elemMatch is spot-on how the validator works out of the box when encountering an array, couldn't that operator be used if you explicitly wanted that behaviour, and then let behaviour without the "rule-plus-doubly-negated-rule" trick work as with it? Perhaps it's just me, but it seems like one would very rarely want to write a "there should be at least one item satisfying my rule if it's an array, otherwise if it's not an array, it should just enforce it" rule. Well, either way, I'm very grateful for your help here! I was a bit disillusioned about not being able to validate arrays properly, and that trick of yours is quite brilliant Cheers! | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 06/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Daniel Smedegaard Buus, Thank you for the additional information. My apologies for providing incorrect information earlier. You are correct that the $type operator matches the elements of the array, however it will match if any element in the array matches the type, even if not all do:
To make sure that all elements in the array are objects, you can add another expression to the $and portion of the validator:
The $elemMatch expression will match all elements that are not objects, and the preceding $not will invert the match and cause the $and to fail if any elements that are not objects are matched. I've verified this with your examples and it looks to work as desired:
Thanks, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Smedegaard Buus [X] [ 06/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
FYI, the issue is reproducible without the whole $or part of the validator, a simple
is enough to reproduce it. With that validator, once you have a document with a string in a .strings array property, you can $push anything into that array. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Smedegaard Buus [X] [ 06/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Mark Thanks for your response. I'm a bit puzzled by your answer, though, because the $type operator, when applied to an array member is documented, as in that link, "When applied to arrays, $type matches any inner element that is of the specified BSON type. For example, when matching for $type : 'array', the document will match if the field has a nested array. It will not return results where the field itself is an array." Also, the actual behavior does not correspond with your suggestion, since then all of my insert examples offering an array for a dbRefsMember value should validate, which they do not. This further doesn't correspond with the fact that I'm allowed to push both strings, integers and booleans into an existing dbRefsMember array, but not an object that violates the validation rule? Actually, I felt like I was rubberducking while writing the above and had a thought. I did a little testing, and here's the exact same problem, with the object suggestion removed from the equation:
Okay, clearly document validation is supposed to work on array items, because it does so with inserts, and with the $push update example in my original example where an object failed the "no ._id" rule. It's almost as if the validation logic works by using the validator as query argument to find() — at least that matches every example here. If you imagine each and every attempted insert and update in my examples was applied without a validator, and then you issued a find() with the document validation rule a query parameter, then you can see that every time that find would have returned the document (because find({strings:{$type:'string')}} matches when there is a string in an array, not when there is only strings — and similarly for the previous _id rule), the update or insert with the validator on validates, and every time it would not, it rejects. Clearly this is not just me being dim, is it? Cheers, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 05/Oct/17 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Daniel Smedegaard Buus, Thank you for the report. Looking over the reproduction steps, I believe the issue may be the
therefore as long as _id is not present as an element, and dbRefsMember is an object, it will pass validation, regardless of the type of the elements in dbRefsMember. Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group. Thanks, |