[SERVER-36070] Aggregation with $out results in error when Auditing is enabled Created: 11/Jul/18  Updated: 29/Oct/23  Resolved: 02/Aug/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Security
Affects Version/s: 3.6.2
Fix Version/s: 3.6.7, 4.0.2, 4.1.2

Type: Bug Priority: Critical - P2
Reporter: Harshad Dhavale Assignee: Spencer Jackson
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-36606 Remove size limits on BSON audit events Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6
Sprint: Platforms 2018-08-13
Participants:
Case:

 Description   

MongoDB version 3.6.

When auditing is enabled, aggregation query with $out, fails with the error "Size must be between 0 and 16793600(16MB) error" even though there is no document in the collection that is greater than 16MB. 

For example, with auditing enabled on mongod, when the following query is executed:

  

db.collection.aggregate([ {$match:{}}, {$out:"collection2"} ],{allowDiskUse:true})

 

it throws this error:

 

assert: command failed: { "operationTime" : Timestamp(1529677911, 2), "ok" : 0, "errmsg" : "insert for $out failed: { lastOp: { ts: Timestamp(1529677911, 1), t: 4 } , connectionId: 202, err: \"BSONObj size: 16793954 (0x1004162) is invalid. Size must be between 0 and 16793600(16MB) First element: : { _id: ObjectId('5b2854bfee28fa281454cb45'),...\", code: 10334, codeName: \"Location10334\", n: 0, ok: 1.0, operationTime: Timestamp(1529677911, 1), $gleStats: { lastOpTime: { ts: Timestamp(1529677911, 1), t: 4 } , electionId: ObjectId('7fffffff0000000000000004') }, $clusterTime: { clusterTime: Timestamp(1529677911, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, $configServerState: { opTime: { ts:

 

 
 
When auditing is not enabled, the aggregation query succeeds.



 Comments   
Comment by Githook User [ 09/Aug/18 ]

Author:

{'username': 'spencerjackson', 'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com'}

Message: SERVER-36070: Generate audit BSONObjs which match serialization logic

(cherry picked from commit 0cb1a69acc5ce7a67a4241e75d23b2f5b68fafe0)
Branch: v4.0
https://github.com/10gen/mongo-enterprise-modules/commit/9a5e6ac46c57826c983373bb027539da3af95c00

Comment by Harshad Dhavale [ 03/Aug/18 ]

Thanks spencer.jackson! I really appreciate it!

Comment by Spencer Jackson [ 03/Aug/18 ]

Hi harshad.dhavale, I have completed the backport from master to the 3.6 branch, which will include the patch in that branch's next scheduled release, 3.6.7.

For context, we perform development work on the master branch of the server, which we then backport onto the stable branches. We close the SERVER ticket once the work has landed in master to indicate that the hands-on-keyboards portion of the task has been completed, and manage the backports through Jira metadata.

Comment by Githook User [ 03/Aug/18 ]

Author:

{'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com', 'username': 'spencerjackson'}

Message: SERVER-36070: Generate audit BSONObjs which match serialization logic

(cherry picked from commit 0cb1a69acc5ce7a67a4241e75d23b2f5b68fafe0)
Branch: v3.6
https://github.com/10gen/mongo-enterprise-modules/commit/be82936d28717f516d18bacb69972cfa36838a8a

Comment by Harshad Dhavale [ 03/Aug/18 ]

Hi spencer.jackson ,

Thanks for the updates on this ticket. I see that the Jira has now been closed and marked as fixed in version 4.1.2. Please could you share the details of which 3.6 version will the fix be backported to, and when is the approximate date for that specific 3.6 version to be released?

Thanks again!

Regards,
Harshad 

Comment by Githook User [ 02/Aug/18 ]

Author:

{'username': 'spencerjackson', 'name': 'Spencer Jackson', 'email': 'spencer.jackson@mongodb.com'}

Message: SERVER-36070: Generate audit BSONObjs which match serialization logic
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/0cb1a69acc5ce7a67a4241e75d23b2f5b68fafe0

Comment by Andrew Morrow (Inactive) [ 29/Jul/18 ]

harshad.dhavale - Well, so far we just have a hypothesis about what went wrong - no fix has been identified yet. I'm sure spencer.jackson will reach out with additional information as it becomes available, but I wouldn't think it reasonable to provide any sort of estimate for a fix yet.

Comment by Harshad Dhavale [ 24/Jul/18 ]

Hi spencer.jackson - Just want to check for your and the platforms team's assessment of this issue. Would really appreciate it if you could share any findings/assessment/diagnosis/eta information on it. Thanks.

Comment by Harshad Dhavale [ 23/Jul/18 ]

Hi acm - just wanted to followup and check whether there were any findings/diagnosis/possible-ETA for this issue, in the platforms leads meeting today.

Thanks.

Comment by Harshad Dhavale [ 18/Jul/18 ]

Thanks acm for prioritizing this ticket! I will look forward to the platform team's findings/diagnosis, and if possible an ETA for the fix.

Comment by Andrew Morrow (Inactive) [ 18/Jul/18 ]

harshad.dhavale - We will be triaging this ticket at the Monday platforms leads meeting.

Comment by Asya Kamsky [ 13/Jul/18 ]

Since this is auditing should be triaged by security/platforms team. 

Comment by Asya Kamsky [ 13/Jul/18 ]

It doesn’t always fail though - only on certain combinations of document sizes. 

 

And it didnt fail in 3.2 

Generated at Thu Feb 08 04:41:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.