[JAVA-66] Inserting very large java.util.Collections into DBCollection causes a buffer overflow exception Created: 20/Dec/09  Updated: 13/Jan/10  Resolved: 21/Dec/09

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: 1.1, 1.2
Fix Version/s: 1.2

Type: Bug Priority: Major - P3
Reporter: Phil Messenger Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

When inserting a very long list of DBObjects, a bufferoverflow exception is thrown:

Exception in thread "main" java.lang.IllegalArgumentException: tried to save too large of an object.  max size : 2098176
	at com.mongodb.ByteEncoder.putObject(ByteEncoder.java:175)
	at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:203)
	at com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:162)
	at tv.miniweb.mediasearch.indexer.Indexer.Index(Indexer.java:54)
	at tv.miniweb.mediasearch.indexer.IndexerLauncher.main(IndexerLauncher.java:53)

This is because a single ByteEncoder is used for the entire .insert request, which is IMO not optimal behaviour in this case. I have pushed a git changeset with a patch and unit tests that fix this.



 Comments   
Comment by Phil Messenger [ 21/Dec/09 ]

Smart, knew I was missing something. Thanks for that!

Comment by Eliot Horowitz (Inactive) [ 21/Dec/09 ]

Not quite.
when writing to the buffer, limit is how far you can write.

From javadoc on flip:
Flips this buffer. The limit is set to the current position and then the position is set to zero. If the mark is defined then it is discarded.
http://java.sun.com/j2se/1.4.2/docs/api/java/nio/Buffer.html#flip()

So, it sends from 0 to position.

Comment by Phil Messenger [ 21/Dec/09 ]

Yup, I see the position( ... ) call in DBAPILayer, but AFAIK that doesn't adjust the limit - so immediately after this it calls .flip(), which is just a delegate through to the underlying bytebuffers flip() method. doInsert( ... ) is then called, which eventually falls through to SocketChannel.write( ), which writes up to the limit() of the original byte buffer, including either the duff data or a load of blanks (which might be a performance concern?).

Mind you, I don't know this code well so I could be missing something really obvious

Comment by Eliot Horowitz (Inactive) [ 21/Dec/09 ]

its not - see the change to DBAPiLayer - it moves the position back on the ByteBuffer
though it wouldn't be a problem anyway - just a warning message on the server.

Comment by Phil Messenger [ 21/Dec/09 ]

Cool, thanks for fixing. I assume it's not a problem that a chunk of data from the last-but-failed object is written to the mongod instance (guess not, the integration test passes)?

Comment by Eliot Horowitz (Inactive) [ 21/Dec/09 ]

fixed it a bit differently, but fixed nonetheless.

Comment by auto [ 21/Dec/09 ]

Author:

{'name': 'Eliot Horowitz', 'email': 'eliot@10gen.com'}

Message: if a bulk insert is too big for 1 buffer, break it up JAVA-66
http://github.com/mongodb/mongo-java-driver/commit/76ccc51dea8453f48e3303ffeef40c761244e501

Comment by Phil Messenger [ 20/Dec/09 ]

For reference, the push in question is

2824bead0ade2deb90235a1d39f0f37699f08ba at git://github.com/philmes/mongo-java-driver.git (see http://github.com/philmes/mongo-java-driver/commit/32824bead0ade2deb90235a1d39f0f37699f08ba)

Generated at Thu Feb 08 08:51:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.