[SERVER-54710] Large number of $or clauses can create profiling entry exceeding max BSON size, causing the query to fail when it should not Created: 22/Feb/21 Updated: 29/Oct/23 Resolved: 11/Mar/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | None |
| Fix Version/s: | 4.9.0, 4.4.5, 4.0.24, 4.2.14 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Jennifer Huang (Inactive) | Assignee: | David Storch |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Backport Requested: |
v4.4, v4.2, v4.0
|
||||||||||||
| Sprint: | Query Execution 2021-03-22 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Each clause in a $or expression can use its own index so when a query has a lot $or and each $or has a lot of clause it can create a very large profiler entry when profiler is enabled.
When the profiler entry is too large it can trigger the "BSONObjectTooLarge" error and causing such a query fail to execute.
|
| Comments |
| Comment by Githook User [ 26/Mar/21 ] | ||||||||||||||||
|
Author: {'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}Message: (cherry picked from commit 4fc3991bf64b33ca5f5237722bc563f8eb1a552a) | ||||||||||||||||
| Comment by Githook User [ 24/Mar/21 ] | ||||||||||||||||
|
Author: {'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}Message: (cherry picked from commit 4fc3991bf64b33ca5f5237722bc563f8eb1a552a) | ||||||||||||||||
| Comment by Githook User [ 24/Mar/21 ] | ||||||||||||||||
|
Author: {'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}Message: (cherry picked from commit 4fc3991bf64b33ca5f5237722bc563f8eb1a552a) | ||||||||||||||||
| Comment by Githook User [ 11/Mar/21 ] | ||||||||||||||||
|
Author: {'name': 'David Storch', 'email': 'david.storch@mongodb.com', 'username': 'dstorch'}Message: | ||||||||||||||||
| Comment by David Storch [ 24/Feb/21 ] | ||||||||||||||||
|
Yeah, if we go with the quick fix I demonstrated above it is definitely safe to backport. | ||||||||||||||||
| Comment by Jennifer Huang (Inactive) [ 24/Feb/21 ] | ||||||||||||||||
|
Thanks david.storch for the detailed insight, can this be backported to older version like | ||||||||||||||||
| Comment by David Storch [ 23/Feb/21 ] | ||||||||||||||||
|
I dug into the root cause of this a bit and have some results to report. The code originally added in This bug reports a scenario in which the aforementioned size check does not kick in correctly. It does not kick in because there is a code path that creates a separate buffer instead of always appending to the original buffer. Therefore, the stats are split across two buffers and we never notice that we have surpassed the 10MB limit. Here is a patch to correct this particular problem, which is probably good enough to resolve this ticket:
A few additional notes, however. First, by default the system.profile collection is sized at 1MB. Therefore, if I run the repro script after my fix, the operation will succeed but it will not be able to write an entry to the system.profile collection because the entry exceeds 1MB. This is noted in the logs with a message like the following:
This isn't necessarily a problem, however, since the query as a whole succeeds as expected. Also, if the user re-sizes system.profile to allow more profiler data, then the profile entry will get written as expected. Second, the fix proposed above works in practice although theoretically it would still be possible for queries to fail with the same symptom. I can think of two reasons for this:
Both of these scenarios are slightly farfetched and seem unlikely to occur in practice. Therefore, my recommendation would be to pursue the quick fix in my patch above. |