[SERVER-36135] Aggregation size limit Created: 14/Jul/18  Updated: 04/Sep/18  Resolved: 18/Jul/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: PB [X] Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Operating System: ALL
Participants:

 Description   

Hi,

Currently I'm working on a Mongo DB. I have a collection with hundreds of millions of records.

 

When I try to run an aggregation I always get the 16Mb size limit error.

 

I'm using Spring and MongoTemplate. 

 

I've google around and tried and used cursor() , out(), allowDiskUse(true) but nothing seems to solve the limit size error.

 

 



 Comments   
Comment by PL [X] [ 24/Jul/18 ]

Hi Nick, 

 

I tried MongoCursor , but I still get the same error:

 

Hi, 

 

When I run The below code I get size limit exception.

 

 

 

 

 

MongoClient mongoClient = new MongoClient(host, port);

MongoCollection<Document> col = mongoClient.getDatabase(db).getCollection(collection;

 

int offset = 0;

int limit = 10000;

BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));

Set<String> set= new HashSet<>();

while (offset < totalCount) {

        List<Document> documents =                 col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>());

         for (Document doc : documents) {

               set.add(doc.getString(field));

            }

                          offset += limit;

   }

//process last 

List<Document> documents = col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>());

        for (Document doc : documents) {

               set.add(doc.getString(field));

            }
mongoClient.close();

 

 

When this line of code " col.find(whereQuery).skip(offset).limit(limit).into(new ArrayList<Document>()); " runs I get this error:

 

 

org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is larger than maximum of 16793600. at org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110) ~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122) ~[mongo-java-driver-3.8.0.jar:na]

 

 As this issue is closed I opened a new issue SERVER-36252

 

 

Comment by Nick Brewer [ 18/Jul/18 ]

MongoDB2018PB,

From what I can tell from the outputs you've provided, it doesn't appear that you're using a cursor - I believe the DBCursor class is what you're looking for.

From your responses, I don't see anything that indicates an underlying bug in the MongoDB server. I can see that you've posted this question on the mongodb-user group, which is a better venue for questions of this nature. 

Regards,
Nick

Comment by PL [X] [ 16/Jul/18 ]

Error stacktrace after upgrading versions to :

 

MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and mongo-java-driver: 3.8.0
ERROR:
 
 
 ERROR:
 
org.bson.BsonMaximumSizeExceededException: Document size of 17543853 is larger than maximum of 16793600.
at org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeIterable(DocumentCodec.java:209) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:180) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:182) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:182) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:182) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeIterable(DocumentCodec.java:207) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeValue(DocumentCodec.java:180) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.writeMap(DocumentCodec.java:199) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:141) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.DocumentCodec.encode(DocumentCodec.java:45) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63) ~[mongo-java-driver-3.8.0.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:213) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:205) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:115) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:108) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.client.internal.MongoDatabaseImpl.executeCommand(MongoDatabaseImpl.java:182) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.client.internal.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:151) ~[mongo-java-driver-3.8.0.jar:na]
at com.mongodb.client.internal.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:146) ~[mongo-java-driver-3.8.0.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate$4.doInDB(MongoTemplate.java:440) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate$4.doInDB(MongoTemplate.java:437) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:497) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:437) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate$BatchAggregationLoader.aggregateBatched(MongoTemplate.java:3148) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate$BatchAggregationLoader.aggregate(MongoTemplate.java:3122) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1938) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1841) ~[spring-data-mongodb-2.0.8.RELEASE.jar:2.0.8.RELEASE]
 
 

Comment by PL [X] [ 16/Jul/18 ]

Hi Nick,

 

MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and mongo-java-driver: 3.8.0

 Spring version  5.0.7.RELEASE

 

Collection stats:

 

{
"ns" : "test",
"size" : 43280104343.0,
"count" : 141452113,
"avgObjSize" : 305,
"storageSize" : 7562801152.0,
"capped" : false,
"wiredTiger" : {
"metadata" :

{ "formatVersion" : 1 }

,
"creationString" : "access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=true),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u",
"type" : "file",
"uri" : "statistics:table:collection-0--515929282308843229",
"LSM" :

{ "bloom filter false positives" : 0, "bloom filter hits" : 0, "bloom filter misses" : 0, "bloom filter pages evicted from cache" : 0, "bloom filter pages read into cache" : 0, "bloom filters in the LSM tree" : 0, "chunks in the LSM tree" : 0, "highest merge generation in the LSM tree" : 0, "queries that could have benefited from a Bloom filter that did not exist" : 0, "sleep for LSM checkpoint throttle" : 0, "sleep for LSM merge throttle" : 0, "total size of bloom filters" : 0 }

,
"block-manager" :

{ "allocations requiring file extension" : 1840489, "blocks allocated" : 2104282, "blocks freed" : 448592, "checkpoint size" : 6803996672.0, "file allocation unit size" : 4096, "file bytes available for reuse" : 758718464, "file magic number" : 120897, "file major version number" : 1, "file size in bytes" : 7562801152.0, "minor version number" : 0 }

,
"btree" :

{ "btree checkpoint generation" : 214, "column-store fixed-size leaf pages" : 0, "column-store internal pages" : 0, "column-store variable-size RLE encoded values" : 0, "column-store variable-size deleted values" : 0, "column-store variable-size leaf pages" : 0, "fixed-record size" : 0, "maximum internal page key size" : 368, "maximum internal page size" : 4096, "maximum leaf page key size" : 2867, "maximum leaf page size" : 32768, "maximum leaf page value size" : 67108864, "maximum tree depth" : 5, "number of key/value pairs" : 0, "overflow pages" : 0, "pages rewritten by compaction" : 0, "row-store internal pages" : 0, "row-store leaf pages" : 0 }

,
"cache" :

{ "bytes currently in the cache" : 16150829, "bytes read into cache" : 63428160930.0, "bytes written from cache" : 56038149414.0, "checkpoint blocked page eviction" : 830, "data source pages selected for eviction unable to be evicted" : 52028, "eviction walk passes of a file" : 605224, "eviction walk target pages histogram - 0-9" : 21210, "eviction walk target pages histogram - 10-31" : 83139, "eviction walk target pages histogram - 128 and higher" : 0, "eviction walk target pages histogram - 32-63" : 53685, "eviction walk target pages histogram - 64-128" : 447190, "eviction walks abandoned" : 2950, "eviction walks gave up because they restarted their walk twice" : 56, "eviction walks gave up because they saw too many pages and found no candidates" : 11166, "eviction walks gave up because they saw too many pages and found too few candidates" : 4369, "eviction walks reached end of tree" : 16902, "eviction walks started from root of tree" : 18548, "eviction walks started from saved location in tree" : 586676, "hazard pointer blocked page eviction" : 337, "in-memory page passed criteria to be split" : 18487, "in-memory page splits" : 7836, "internal pages evicted" : 37452, "internal pages split during eviction" : 186, "leaf pages split during eviction" : 24125, "modified pages evicted" : 737460, "overflow pages read into cache" : 0, "page split during eviction deepened the tree" : 2, "page written requiring lookaside records" : 32412, "pages read into cache" : 2348580, "pages read into cache after truncate" : 1, "pages read into cache after truncate in prepare state" : 0, "pages read into cache requiring lookaside entries" : 48493, "pages requested from the cache" : 698844111, "pages seen by eviction walk" : 104673298, "pages written from cache" : 2103935, "pages written requiring in-memory restoration" : 247135, "tracked dirty bytes in the cache" : 0, "unmodified pages evicted" : 2895469 }

,
"cache_walk" :

{ "Average difference between current eviction generation when the page was last considered" : 0, "Average on-disk page image size seen" : 0, "Average time in cache for pages that have been visited by the eviction server" : 0, "Average time in cache for pages that have not been visited by the eviction server" : 0, "Clean pages currently in cache" : 0, "Current eviction generation" : 0, "Dirty pages currently in cache" : 0, "Entries in the root page" : 0, "Internal pages currently in cache" : 0, "Leaf pages currently in cache" : 0, "Maximum difference between current eviction generation when the page was last considered" : 0, "Maximum page size seen" : 0, "Minimum on-disk page image size seen" : 0, "Number of pages never visited by eviction server" : 0, "On-disk page image sizes smaller than a single allocation unit" : 0, "Pages created in memory and never written" : 0, "Pages currently queued for eviction" : 0, "Pages that could not be queued for eviction" : 0, "Refs skipped during cache traversal" : 0, "Size of the root page" : 0, "Total number of pages currently in cache" : 0 }

,
"compression" :

{ "compressed pages read" : 2287461, "compressed pages written" : 2037673, "page written failed to compress" : 0, "page written was too small to compress" : 65663, "raw compression call failed, additional data available" : 0, "raw compression call failed, no additional data available" : 0, "raw compression call succeeded" : 0 }

,
"cursor" :

{ "bulk-loaded cursor-insert calls" : 0, "create calls" : 11, "cursor-insert key and value bytes inserted" : 54185049121.0, "cursor-remove key bytes removed" : 152538960, "cursor-update value bytes updated" : 0, "cursors cached on close" : 0, "cursors reused from cache" : 2736470, "insert calls" : 171617646, "modify calls" : 0, "next calls" : 179207188, "prev calls" : 1, "remove calls" : 30507792, "reserve calls" : 0, "reset calls" : 114330540, "restarted searches" : 259596, "search calls" : 91609805, "search near calls" : 1414121, "truncate calls" : 0, "update calls" : 0 }

,
"reconciliation" :

{ "dictionary matches" : 0, "fast-path pages deleted" : 0, "internal page key bytes discarded using suffix compression" : 3921839, "internal page multi-block writes" : 453, "internal-page overflow keys" : 0, "leaf page key bytes discarded using prefix compression" : 0, "leaf page multi-block writes" : 24075, "leaf-page overflow keys" : 0, "maximum blocks required for a page" : 1, "overflow values written" : 0, "page checksum matches" : 18068, "page reconciliation calls" : 890411, "page reconciliation calls for eviction" : 744316, "pages deleted" : 421721 }

,
"session" :

{ "cached cursor count" : 2462, "object compaction" : 0, "open cursor count" : 0 }

,
"transaction" :

{ "update conflicts" : 0 }

},
"nindexes" : 8,
"totalIndexSize" : 10518695936.0,
"indexSizes" :

{ "_id_" : 2049613824.0, "sched_fk_keys" : 3078926336.0, "deliveryId_keys" : 687837184, "locationId_keys" : 988405760, "deliveryId" : 652734464, "locationId" : 1006055424, "psv_fk_keys" : 538570752, "count_keys" : 1516552192.0 }

,
"ok" : 1.0
}

Comment by Nick Brewer [ 16/Jul/18 ]

Hi MngDB2018P

Just to confirm, what version of Spring are you using? Support for aggregation cursors was introduced in 2.0.

Thanks,
Nick

Comment by PB [X] [ 14/Jul/18 ]

 
 
Aggregation agg = newAggregation(
                match(Criteria
                        .where(FIELD_1)
                        .nin(LIST)
                        .and(FIELD_2)
                        .is(VALUE_2))
                        .and(FIELD_3)
                        .is(VALUE_3)
                        .and(FIELD_4)
                        .is(VALUE_4)),
                out(OUT_COL)
 
        ).withOptions(
                new AggregationOptions(true, false, new BasicDBObject()));
 
        mongoTemplate.aggregate(agg, COLLECTION, SOMECLASS.class);
 
 
 ERROR:
 
org.bson.BsonSerializationException: Document size of 17543853 is larger than maximum of 16793600.
at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:376) ~[bson-3.4.3.jar:na]
at org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:143) ~[bson-3.4.3.jar:na]
at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:347) ~[bson-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encodeIterable(DBObjectCodec.java:273) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:200) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:222) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encodeIterable(DBObjectCodec.java:271) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:200) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:130) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:61) ~[mongodb-driver-3.4.3.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63) ~[bson-3.4.3.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[bson-3.4.3.jar:na]
at com.mongodb.connection.RequestMessage.addDocument(RequestMessage.java:253) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.RequestMessage.addDocument(RequestMessage.java:205) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:75) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.RequestMessage.encodeWithMetadata(RequestMessage.java:160) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.CommandProtocol.sendMessage(CommandProtocol.java:186) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:111) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:187) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:179) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:92) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:85) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55) ~[mongodb-driver-core-3.4.3.jar:na]
at com.mongodb.Mongo.execute(Mongo.java:836) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.Mongo$2.execute(Mongo.java:823) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DB.executeCommand(DB.java:729) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DB.command(DB.java:491) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DB.command(DB.java:507) ~[mongodb-driver-3.4.3.jar:na]
at com.mongodb.DB.command(DB.java:462) ~[mongodb-driver-3.4.3.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate$3.doInDB(MongoTemplate.java:410) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate$3.doInDB(MongoTemplate.java:408) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:466) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:408) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate$BatchAggregationLoader.aggregateBatched(MongoTemplate.java:2616) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate$BatchAggregationLoader.aggregate(MongoTemplate.java:2592) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1570) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(MongoTemplate.java:1511) ~[spring-data-mongodb-1.10.10.RELEASE.jar:na]
at pt.brisa.linkbeyond.gtfs.extended.model.parse.delivery.ScheduleService.getCountOfUnrefDelSch(ScheduleService.java:158) ~[gtfs-extended-parser-0.0.1-SNAPSHOT.jar:na]
at pt.brisa.linkbeyond.gtfs.extended.model.parse.delivery.ScheduleService.checkReferentialIntegrity(ScheduleService.java:298) ~[gtfs-extended-parser-0.0.1-SNAPSHOT.jar:na]
at pt.brisa.extended.gtfs.handler.implementation.GtfsService.moveToCache(GtfsService.java:495) ~[gtfs-extended-parser-0.0.1-SNAPSHOT.jar:na]
at pt.brisa.linkbeyond.gtfs.GtfsParserService.parseZipGtfsFile(GtfsParserService.java:2249) ~[gtfs-extended-parser-0.0.1-SNAPSHOT.jar:na]
at pt.brisa.linkbeyond.gtfs.GtfsExtendedController.handle(GtfsExtendedController.java:99) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) [spring-boot-1.5.10.RELEASE.jar:1.5.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.10.RELEASE.jar:1.5.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) [spring-boot-actuator-1.5.10.RELEASE.jar:1.5.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.27.jar:8.5.27]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.27.jar:8.5.27]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_152]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.27.jar:8.5.27]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]

 
 

 
Generated at Thu Feb 08 04:42:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.