[SERVER-44497] Aggregation: {"codeName" : "UnknownError"} Created: 08/Nov/19  Updated: 29/Mar/20  Resolved: 29/Mar/20

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Dheeraj Gunda Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-43151 Error in aggregation assertion at val... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

I have index on "claimId" field, I am aggregating to find duplicate values on "claimId" field. Aggregation with below query is working on Lower Environments (2Million documents). Failing with error on PROD enviroment (500 Million documents)

 

Aggregation Query:

db.CLAIMS_ENTITY_COMM.aggregate([
     { "$group": {
         "_id": { "claimId": "$claimId" },
         "dups": { "$push": "$_id" },
         "count": { "$sum": 1 }
     }},
     { "$match": { "count": { "$gt": 1 } }},
 {$out: "CLAIMS_ENTITY_COMM_DUPS"}
 ],{allowDiskUse : true}).forEach(function(doc) {
     doc.dups.shift(); 
     db.CLAIMS_ENTITY_COMM_DUPS.insert(doc);
 });

Response with error:

2019-11-02T05:22:33.799-0400 E QUERY [js] Error: command failed: {
 "ok" : 0,
 "errmsg" : "assertion src/mongo/db/pipeline/value.cpp:1344",
 "code" : 8,
 "codeName" : "UnknownError",
 "operationTime" : Timestamp(1572686553, 5),
 "$clusterTime" : {
 "clusterTime" : Timestamp(1572686553, 5),
 "signature" : {
 "hash" : BinData(0,"EojyCQa6HKeiJIetOsbAZcljbcM="),
 "keyId" : NumberLong("6704370801467981855")
 }
 }
} : aggregate failed :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
doassert@src/mongo/shell/assert.js:18:14
_assertCommandWorked@src/mongo/shell/assert.js:536:17
assert.commandWorked@src/mongo/shell/assert.js:620:16
DB.prototype._runAggregate@src/mongo/shell/db.js:260:9
DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1062:12
@(shell):1:1



 Comments   
Comment by Dmitry Agranat [ 29/Mar/20 ]

Hi dgunda@express-scripts.com,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Kelsey Schubert [ 19/Dec/19 ]

Hi dgunda@express-scripts.com,

4.0.14 was released earlier today, would you please upgrade and let us know if it resolves the issue?

Thanks,
Kelsey

Comment by David Storch [ 19/Nov/19 ]

Hi dgunda@express-scripts.com, 4.0.14 has not yet been released. When it is released, the fix for SERVER-43151 will be included. I'll leave it to dmitry.agranat to comment regarding the release timeline.

Best,
Dave

Comment by Dheeraj Gunda [ 18/Nov/19 ]

Hi Bernard/ David,

The latest version I can download is 4.0.13, Can you share the link to download version 4.0.14 for Enterprise version?

FYI, I can't upgrade to 4.2 version due to some dependencies.

Thanks,

Dheeraj

 

Comment by Bernard Gorman [ 11/Nov/19 ]

david.storch, dgunda@express-scripts.com: yes, this appears to be an instance of SERVER-43151. The issue was due to the fact that, if the temporary on-disk file used by the $group with allowDiskUse:true exceeded a certain size, we would incorrectly record the offset into the file; this is why the aggregation succeeds on smaller environments but fails on the larger production system. The issue is fixed in 4.0.14 and should allow the aggregation to succeed.

Comment by Dheeraj Gunda [ 11/Nov/19 ]

Hi Dave, Thanks for the suggestion, I will consider to upgrade to 4.0.14

Comment by David Storch [ 11/Nov/19 ]

I suspect that this is a duplicate of SERVER-43151. Perhaps bernard.gorman could confirm?

dgunda@express-scripts.com, SERVER-43151 has been fixed in version 4.0.14, so I recommend an upgrade to the latest patch release of the 4.0 series when 4.0.14 is released.

Best,
Dave

Comment by Dheeraj Gunda [ 10/Nov/19 ]

Hi Dima,

Currently, MongoDB server version: 4.0.8 (Enterprise).

Unable to reproduce the problem in lower environment (DEV, QA). As suggested I will try to run with traceExceptions parameter.

 

Thanks,

Dheeraj

 

Comment by Dmitry Agranat [ 10/Nov/19 ]

Hi dgunda@express-scripts.com,

Thank you for the report. I've looked at the code value.cpp:1344 but I don't see any assert there.

Would it be possible to run the failing mongoD with the traceExceptions parameter? That would give us more information. Please note that the parameter will potentially generate a lot of logs so it can impact a performance. It would be great if the problem reproduces in a staging/testing environment.

Also, please provide your MongoDB version.

Thanks,
Dima

Comment by Dheeraj Gunda [ 08/Nov/19 ]

Aggregation Query:

 

db.CLAIMS_ENTITY_COMM.aggregate([   
{ "$group": { "_id":

{ "claimId": "$claimId" }

,       
"dups": { "$push": "$_id" },       
"count": { "$sum": 1 }    }},   
{ "$match": { "count":

{ "$gt": 1 }

}},
{$out: "CLAIMS_ENTITY_COMM_DUPS"}],

{allowDiskUse : true}

).forEach(function(doc) {    doc.dups.shift();     db.CLAIMS_ENTITY_COMM_DUPS.insert(doc);});

 

Generated at Thu Feb 08 05:06:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.