[DRIVERS-2137] Add bulk write prose test to ensure that maxMessageSizeBytes is not exceeded Created: 22/Nov/19  Updated: 29/Jun/22

Status: Backlog
Project: Drivers
Component/s: CRUD
Fix Version/s: None

Type: Spec Change Priority: Major - P3
Reporter: Shane Harvey Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Related
related to PYTHON-2055 Large bulk inserts with compressor th... Closed
Driver Changes: Needed

 Description   

In PYTHON-2055, we discovered that PyMongo could generate a batched bulk write larger than maxMessageSizeBytes. Specifically, when using OP_MSG with OP_COMPRESSED we did not factor in the 16-byte message header.

In response I added this test to ensure that batch sizes are calculated appropriately:

    @client_context.require_version_min(3, 6)
    def test_bulk_max_message_size(self):
        self.coll.delete_many({})
        self.addCleanup(self.coll.delete_many, {})
        _16_MB = 16 * 1000 * 1000
        # Generate a list of documents such that the first batched OP_MSG is
        # as close as possible to the 48MB limit.
        docs = [
            {'_id': 1, 'l': 's' * _16_MB},
            {'_id': 2, 'l': 's' * _16_MB},
            {'_id': 3, 'l': 's' * (_16_MB - 10000)},
        ]
        # Fill in the remaining ~10000 bytes with small documents.
        for i in range(4, 10000):
            docs.append({'_id': i})
        result = self.coll.insert_many(docs)
        self.assertEqual(len(docs), len(result.inserted_ids))

I propose we turn this into a prose test to help avoid this bug in other drivers.



 Comments   
Comment by Jeremy Mikola [ 13/Jan/20 ]

As we're moving this to the backlog, I gather that this is most easily implemented as a prose test since we have no spec test syntax to do match for these calculations. You're welcome to explore doing this in a spec test, but I think prose would suffice.

Comment by Shane Harvey [ 22/Nov/19 ]

Agreed, but I think this test is useful without compression too and I want to avoid putting this test in two places. Perhaps we can update the compression spec to require that drivers run all the bulk write/CRUD tests with compression enabled (or even better the entire test suite).

Comment by Bernie Hackett [ 22/Nov/19 ]

This seems like a compression test to me. It requires wire compression, not just bulk operations.

Generated at Thu Feb 08 08:24:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.