[SERVER-5849] Miessage max size limit Created: 16/May/12  Updated: 01/Nov/12  Resolved: 16/Jun/12

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 2.0.2
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Maxence Dalmais Assignee: Christian Amor Kvalheim
Resolution: Done Votes: 0
Labels: message, size
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Hi,

playing with mongo and sending big bulks insert commands, I get the error

"Wed May 16 11:59:01 [conn139] recv(): message len 58721588 is too large58721588"

Looking at the code about this limitation, I found the limit being 48000000 and defined in message_port.cpp

Can you explain me why this limit of 48000000 came from ?
It is just an arbitrary limit defined as approximately 3 times the maximum document size ?

I found nowhere else in the code a reference to this number, except in js-1.7 which severals times use the number 0x48000000.
Since I found no others reason for the 48000000 limits, I was wondering that maybe this limit should be 0x48000000 instead of 4800000.

Does somebody have some insight about this ?



 Comments   
Comment by Roger Binns [ 01/Nov/12 ]

This random limit causes a lot of grief for bulk inserts and the resulting behaviour in the client (pymongo in my case) means you just get an abruptly dropped connection and no further explanation.

Comment by Christian Amor Kvalheim [ 17/May/12 ]

github reference issue https://github.com/mongodb/node-mongodb-native/issues/609

Comment by Christian Amor Kvalheim [ 17/May/12 ]

I would suggest you just up the poolsize of the driver and restrict the bulk inserts to the max 16GB and then if you need fork child processes in node. It's very likely you'll flood the tcp connection between the inserter and the server so that will most likely be your upper limit on insertion speed.

Comment by Scott Hernandez (Inactive) [ 17/May/12 ]

The limit on the server is an extreme, and there if drivers make simple mistakes. Drivers are basically written to use the max doc size plus some overhead.

Comment by Maxence Dalmais [ 17/May/12 ]

I used a modify version of the node-mongodb-native for testing.

The fact is that this limit is not documented, and cannot be read remotely by a client. (Whereas maxBsonObjectSize is)

The official node-mongodb-native implementation has a message limit of 16MB(+headers I guess).
Should it use this 48000000 limit ?
Would it works with old mongo version, or maybe the previous limit was 3*4MB ?

Comment by Eliot Horowitz (Inactive) [ 17/May/12 ]

The limit is as you said, roughly 3x max doc size.

what client are you using?
They shouldn't be generating a bulk insert message that large.

Generated at Thu Feb 08 03:10:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.