[SERVER-66289] $out incorrectly throws BSONObj size error on v5.0.8 Created: 06/May/22 Updated: 29/Oct/23 Resolved: 16/Aug/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Query Execution, Shell |
| Affects Version/s: | 5.0.8 |
| Fix Version/s: | 4.4.18, 5.0.14, 6.0.3, 6.1.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Awanish Raj | Assignee: | Mihai Andrei |
| Resolution: | Fixed | Votes: | 2 |
| Labels: | $out, aggregation, bug | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Backport Requested: |
v6.0, v5.0, v4.4
|
||||||||||||||||||||||||
| Steps To Reproduce: | Steps followed:
On version 4.4.13: The command runs successfully and movies2 collection is created.
On version 5.0.8: The command fails with the error: PlanExecutor error during aggregation :: caused by :: BSONObj size: 16836845 (0x100E8ED) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "tmp.agg_out.045d5052-e014-4830-8d16-5b985ebfc46e" |
||||||||||||||||||||||||
| Sprint: | QE 2022-05-30, QE 2022-06-13, QE 2022-06-27, QE 2022-07-11, QE 2022-07-25, QE 2022-08-08, QE 2022-08-22 | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||
| Linked BF Score: | 170 | ||||||||||||||||||||||||
| Description |
|
In version 5, aggregation with $out as final stage is throwing BSONObj size (16MB) error even though individual documents are much smaller. It seems to be measuring the total size of all the documents in the result set. |
| Comments |
| Comment by Githook User [ 05/Oct/22 ] | |||||||||||||||
|
Author: {'name': 'Mihai Andrei', 'email': 'mihai.andrei@10gen.com', 'username': 'mtandrei'}Message: (cherry picked from commit 707ba0a0ade42c4540b9cabaaf5a257de944cc3e) | |||||||||||||||
| Comment by Githook User [ 04/Oct/22 ] | |||||||||||||||
|
Author: {'name': 'Mihai Andrei', 'email': 'mihai.andrei@10gen.com', 'username': 'mtandrei'}Message: (cherry picked from commit 707ba0a0ade42c4540b9cabaaf5a257de944cc3e) | |||||||||||||||
| Comment by Githook User [ 04/Oct/22 ] | |||||||||||||||
|
Author: {'name': 'Mihai Andrei', 'email': 'mihai.andrei@10gen.com', 'username': 'mtandrei'}Message: (cherry picked from commit 707ba0a0ade42c4540b9cabaaf5a257de944cc3e) | |||||||||||||||
| Comment by Mihai Andrei [ 03/Oct/22 ] | |||||||||||||||
|
The real commit that wound up in master: https://github.com/mongodb/mongo/commit/707ba0a0ade42c4540b9cabaaf5a257de944cc3e (note that https://github.com/mongodb/mongo/commit/7b7fe658db948e6f5a4a6c30d4590d7866c59371 was reverted) I wonder why the githook didn't pick this one up... oh well | |||||||||||||||
| Comment by Githook User [ 30/Sep/22 ] | |||||||||||||||
|
Author: {'name': 'Uladzimir Makouski', 'email': 'uladzimir.makouski@mongodb.com', 'username': 'umakouski'}Message: Revert " This reverts commit dff46e92439898e1012f93ee7ca82d35ad88dae5. | |||||||||||||||
| Comment by Githook User [ 29/Sep/22 ] | |||||||||||||||
|
Author: {'name': 'Mihai Andrei', 'email': 'mihai.andrei@10gen.com', 'username': 'mtandrei'}Message: (cherry picked from commit 7b7fe658db948e6f5a4a6c30d4590d7866c59371) | |||||||||||||||
| Comment by Lars Van Casteren [ 14/Sep/22 ] | |||||||||||||||
|
Thanks! | |||||||||||||||
| Comment by Kyle Suarez [ 14/Sep/22 ] | |||||||||||||||
|
larsvancasteren@gmail.com, the fix in this ticket is present in 6.1.0-rc0; however, the team has not yet backported the fix to previous branches. We've requested backport to the v6.0, v5.0 and v4.4 versions. Please continue to watch this ticket for updates and thank you for your patience. Kyle | |||||||||||||||
| Comment by Lars Van Casteren [ 14/Sep/22 ] | |||||||||||||||
|
MongoDB 4.4.15 When updating a $out (aggregation) on a primary node without any readPreference it runs successful.
For completness: https://www.mongodb.com/community/forums/t/mongodb-4-4-15-bsonobjecttoolarge-aggregation-out-update/186983 Gr, | |||||||||||||||
| Comment by Githook User [ 27/Jul/22 ] | |||||||||||||||
|
Author: {'name': 'Mihai Andrei', 'email': 'mihai.andrei@10gen.com', 'username': 'mtandrei'}Message: | |||||||||||||||
| Comment by Mihai Andrei [ 01/Jun/22 ] | |||||||||||||||
|
Thanks for filing this ticket and reporting this issue! After doing some investigation, there does appear to be an issue with how the server handles batches of writes when running $out with secondary read preference. One thing I would like to note however, is that when investigating, I was able to reproduce the issue on 4.4 as well as 5.0. Would you be able to confirm that when you ran your aggregation with secondary read preference on 4.4, that the aggregate succeeded? Also, which shell did you use? I would like to rule out the possibility of a separate issue (that of the shell or drivers not respecting read preference). | |||||||||||||||
| Comment by Chris Kelly [ 24/May/22 ] | |||||||||||||||
|
Hello, Given that this issue appears to be behaving inconsistently depending on where the query is ran from, I'm going to forward this to Driver Escalation for further investigation, given that:
Christopher | |||||||||||||||
| Comment by Clayton Taylor [ 12/May/22 ] | |||||||||||||||
|
Seeing this issue as well; a few details from my team's end:
| |||||||||||||||
| Comment by Awanish Raj [ 10/May/22 ] | |||||||||||||||
|
The issue happens only if readPreference was "secondary". If no readPreference is given, the command succeeds. | |||||||||||||||
| Comment by Awanish Raj [ 10/May/22 ] | |||||||||||||||
|
The error pops up when using Realm Triggers (In production) and MongoDB Compass Shell (In testing) to run this command so far. When I ran the same from mongosh in terminal, there was no issue and it succeeded. | |||||||||||||||
| Comment by Awanish Raj [ 06/May/22 ] | |||||||||||||||
|
I have verified that the aggregation succeeds on v4.4.13, but fails on v5.0.8 and v5.3.1. | |||||||||||||||
| Comment by Awanish Raj [ 06/May/22 ] | |||||||||||||||
|
I have tested it with v5.3 on Atlas. The issue remains. |