[SERVER-36135] Aggregation size limit Created: 14/Jul/18 Updated: 04/Sep/18 Resolved: 18/Jul/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | PB [X] | Assignee: | Nick Brewer |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Operating System: | ALL | ||||
| Participants: | |||||
| Description |
|
Hi, Currently I'm working on a Mongo DB. I have a collection with hundreds of millions of records.
When I try to run an aggregation I always get the 16Mb size limit error.
I'm using Spring and MongoTemplate.
I've google around and tried and used cursor() , out(), allowDiskUse(true) but nothing seems to solve the limit size error.
|
| Comments |
| Comment by PL [X] [ 24/Jul/18 ] | ||
|
Hi Nick,
I tried MongoCursor , but I still get the same error:
Hi,
When I run The below code I get size limit exception.
MongoClient mongoClient = new MongoClient(host, port); MongoCollection<Document> col = mongoClient.getDatabase(db).getCollection(collection;
int offset = 0; int limit = 10000; BasicDBObject whereQuery = new BasicDBObject(); Set<String> set= new HashSet<>(); while (offset < totalCount) { List<Document> documents = col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>()); for (Document doc : documents) { set.add(doc.getString(field)); } offset += limit; } //process last List<Document> documents = col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>()); for (Document doc : documents) { set.add(doc.getString(field)); }
When this line of code " col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>()); " runs I get this error:
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is larger than maximum of 16793600. at org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122) ~[mongo-java-driver-3.8.0.jar:na]
As this issue is closed I opened a new issue
| ||
| Comment by Nick Brewer [ 18/Jul/18 ] | ||
|
From what I can tell from the outputs you've provided, it doesn't appear that you're using a cursor - I believe the DBCursor class is what you're looking for. From your responses, I don't see anything that indicates an underlying bug in the MongoDB server. I can see that you've posted this question on the mongodb-user group, which is a better venue for questions of this nature. Regards, | ||
| Comment by PL [X] [ 16/Jul/18 ] | ||
|
Error stacktrace after upgrading versions to :
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and mongo-java-driver: 3.8.0 | ||
| Comment by PL [X] [ 16/Jul/18 ] | ||
|
Hi Nick,
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and mongo-java-driver: 3.8.0 Spring version 5.0.7.RELEASE
Collection stats:
{ , , , , , , , , , , }, , | ||
| Comment by Nick Brewer [ 16/Jul/18 ] | ||
|
Hi MngDB2018P Just to confirm, what version of Spring are you using? Support for aggregation cursors was introduced in 2.0. Thanks, | ||
| Comment by PB [X] [ 14/Jul/18 ] | ||
|
|