[SERVER-6543] Possible to bulk insert more than 16MB of docs but not possible to retry via WBL Created: 20/Jul/12 Updated: 11/Jul/16 Resolved: 25/Oct/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 2.3.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Greg Studer | Assignee: | Greg Studer |
| Resolution: | Done | Votes: | 0 |
| Labels: | continueOnError, getLastError | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
The sharded insert code now correctly groups bulk inserts for different chunks, but the message size limit is larger than the bson size limit when retrying the bulk inserts via a WBL BSON object. We can limit the bulk insert size we send to mongod to 1/2 the max BSON object size, or some other factor, however there's still the edge case when inserting documents very close to 16MB. We can still overrun the actual buffer size in this case due to the extra WBL metadata. |
| Comments |
| Comment by Greg Studer [ 21/Nov/12 ] |
|
After some discussion, the full fix here (depending on |
| Comment by Greg Studer [ 25/Oct/12 ] |
|
Discussing backportability now. |
| Comment by Greg Studer [ 20/Sep/12 ] |
|
Linked to |
| Comment by Greg Studer [ 10/Aug/12 ] |
|
patch in review |
| Comment by Greg Studer [ 20/Jul/12 ] |
|
Second idea - maybe break up writebacks for bulk inserts into several sub-writebacks, avoiding the edge case. Have to be careful this is correct, however. |