Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-1045

Multiprocessing can cause silent fail in insert



    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Works as Designed
    • Affects Version/s: 2.7, 2.8, 3.0, 3.2
    • Fix Version/s: None
    • Component/s: None



      I'm from the Mongoengine team (https://github.com/MongoEngine/mongoengine), back there we have a strange bug in our continuous integration (see for example https://travis-ci.org/MongoEngine/mongoengine/jobs/83760429).
      Basically we create a document with a dict as primary key, but it cannot be found in the database.

      This error occurs when the database is under heavy load (we run 28 concurrent testing environment in travis.ci, I guess they are all running on the same machine with at least 3 of them are running concurrently)
      Furthermore, when I manually rerun the failed environment once the others are done, the error no longer happened.

      Si I wrote a small piece of code to test pymongo support of multiprocess concurrency (see file enclosed).
      The code launches 4 processes, each creating it own MongoClient and insert() 4 empty documents. Finally the number of documents in database is checked (there should be 16 of them).

      In reality, some of the documents are missing in the database (sometime I have 13 documents written, sometime 4 etc.). Interestingly it's always the first inserted documents which are missing (I would have guessed the opposite thinking the latter inserts should be cancelled under too heavy load...).
      I've tested with versions 2.7, 2.8, 3.0 and 3.2, each failing the test.

      I didn't found any information about such a behavior in the documentation and in the existing issues, so here I am


        1. test.py
          0.7 kB



            • Votes:
              0 Vote for this issue
              2 Start watching this issue


              • Created: