Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-2916

Large Bulk batch failures under transactions

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Write Ops
    • Labels:

      Very large Bulk batched write operations inside a transaction fail with transaction conflicts.

      Error from the reproduction script:

      Creating operations
      Starting transaction with 200 operations
      Start session 25e4510e-da67-47c6-9634-4d28242ca3ab
      Synthetic op to start transaction
      Attempting bulk write of 200 operations
      Inside transaction 1
      Synthetic op to start transaction
      Attempting bulk write of 200 operations
      Inside transaction 2
      Synthetic op to start transaction
      ...
      Attempting bulk write of 200 operations
      Inside transaction 164
      Traceback (most recent call last):
        File "bulk_transactions.py", line 84, in <module>
          session.with_transaction(
        File "/usr/local/lib/python3.9/site-packages/pymongo/client_session.py", line 620, in with_transaction
          ret = callback(self)
        File "bulk_transactions.py", line 72, in perform_operation
          res = coll.bulk_write(ops, ordered=False, session=s)
        File "/usr/local/lib/python3.9/site-packages/pymongo/collection.py", line 508, in bulk_write
          bulk_api_result = blk.execute(write_concern, session)
        File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 529, in execute
          return self.execute_command(generator, write_concern, session)
        File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 360, in execute_command
          client._retry_with_session(
        File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1438, in _retry_with_session
          return self._retry_internal(retryable, func, session, bulk)
        File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1470, in _retry_internal
          return func(session, sock_info, retryable)
        File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 354, in retryable_bulk
          self._execute_command(
        File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 310, in _execute_command
          result, to_send = bwc.execute(ops, client)
        File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 950, in execute
          result = self.write_command(request_id, msg, to_send)
        File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 1042, in write_command
          reply = self.sock_info.write_command(request_id, msg)
        File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 798, in write_command
          helpers._check_command_response(result, self.max_wire_version)
        File "/usr/local/lib/python3.9/site-packages/pymongo/helpers.py", line 167, in _check_command_response
          raise OperationFailure(errmsg, code, response, max_wire_version)
      pymongo.errors.OperationFailure: Encountered error from localhost:27018 during a transaction :: caused by :: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction., full error: {'ok': 0.0, 'errmsg': 'Encountered error from localhost:27018 during a transaction :: caused by :: WriteConflict error: this operation conflicted with another operation. Please retry your operation or multi-document transaction.', 'code': 112, 'codeName': 'WriteConflict', 'operationTime': Timestamp(1632204487, 4), '$clusterTime': {'clusterTime': Timestamp(1632204487, 4), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}, 'errorLabels': ['TransientTransactionError']}
      

      Python version: 3.9.7
      pymongo version: 3.12.0
      mongod version: 4.4.6-ent (sharded)

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            luke.prochazka@mongodb.com Luke Prochazka
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: