Details
-
Spec Change
-
Resolution: Unresolved
-
Major - P3
-
None
-
None
-
Needed
Description
In PYTHON-2055, we discovered that PyMongo could generate a batched bulk write larger than maxMessageSizeBytes. Specifically, when using OP_MSG with OP_COMPRESSED we did not factor in the 16-byte message header.
In response I added this test to ensure that batch sizes are calculated appropriately:
@client_context.require_version_min(3, 6) |
def test_bulk_max_message_size(self): |
self.coll.delete_many({}) |
self.addCleanup(self.coll.delete_many, {}) |
_16_MB = 16 * 1000 * 1000 |
# Generate a list of documents such that the first batched OP_MSG is |
# as close as possible to the 48MB limit. |
docs = [ |
{'_id': 1, 'l': 's' * _16_MB}, |
{'_id': 2, 'l': 's' * _16_MB}, |
{'_id': 3, 'l': 's' * (_16_MB - 10000)}, |
]
|
# Fill in the remaining ~10000 bytes with small documents. |
for i in range(4, 10000): |
docs.append({'_id': i}) |
result = self.coll.insert_many(docs) |
self.assertEqual(len(docs), len(result.inserted_ids)) |
I propose we turn this into a prose test to help avoid this bug in other drivers.
Attachments
Issue Links
- related to
-
PYTHON-2055 Large bulk inserts with compressor throws DBException error
-
- Closed
-