Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-67350

stalling during concurrent insertMany operations when unique index exists

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 5.0.9, 4.4.15
    • Component/s: Concurrency
    • Labels:
      None
    • ALL
    • Hide

      I have created a simple java program to try demonstrate the issue

      https://github.com/bobthekingofegypt/mongodb-insert-test

      If you have a separate running mongodb server this can be run from the commandline with 

      ```
      ./gradlew bootRun --args="mongodb://<connection string> <number of connections>"
      ```

      Running it with 5 connections usually demonstrates the problem on my machine and gcp servers.  This program tries to simulate load by just throwing 1 million entries into test database using threads to simulate the concurrency.  We host mongodb ourselves, not currently using clusters for this part of processing so it is a standalone server.

      Show
      I have created a simple java program to try demonstrate the issue https://github.com/bobthekingofegypt/mongodb-insert-test If you have a separate running mongodb server this can be run from the commandline with  ``` ./gradlew bootRun --args="mongodb://<connection string> <number of connections>" ``` Running it with 5 connections usually demonstrates the problem on my machine and gcp servers.  This program tries to simulate load by just throwing 1 million entries into test database using threads to simulate the concurrency.  We host mongodb ourselves, not currently using clusters for this part of processing so it is a standalone server.
    • Execution Team 2022-10-17

      I was doing some testing on the data loading part of our pipeline and have been noticing a lot of problems when multiple concurrent insertMany operations are being triggered on a collection that contains a unique index.  For some reason MongoDB will just stop in the middle of the operations and the socket will stall, Mongodb will be using one or two cores at 100% CPU but there will be nothing in the logs, mongotop, currentOps or mongostat to show that anything is happening.  Eventually the connections will resume and finish the inserts without any errors but this may take anything from 2-10 minutes on my computer (my colleagues have tested it on their slower macbooks and have had stalling for up to 40 minutes).  In case it helps I don't believe the collection is locked, I can start another insert on another terminal with 1 connection and it will complete an insert of 1million rows without issue even when the original sockets are still stuck.

      I have tested using docker to install a few different versions and this problem does not happen on v4.2, but does on v4.4 and v5.0.  On version v4.2 the test program will complete in 1.7 seconds where on the last v5 run I did it took 9m19sec.

      I don't know if I'm doing something wrong, I should tweak some server configuration or if this is expected but it seems extreme.

        1. image-2022-06-27-03-40-25-783.png
          image-2022-06-27-03-40-25-783.png
          134 kB
        2. image-2022-06-27-03-44-24-766.png
          image-2022-06-27-03-44-24-766.png
          139 kB
        3. image-2022-06-27-03-45-22-122.png
          image-2022-06-27-03-45-22-122.png
          135 kB
        4. image-2022-06-27-03-52-55-599.png
          image-2022-06-27-03-52-55-599.png
          143 kB
        5. image-2022-06-27-04-09-41-371.png
          image-2022-06-27-04-09-41-371.png
          142 kB
        6. image-2022-07-14-08-36-37-230.png
          image-2022-07-14-08-36-37-230.png
          84 kB
        7. image-2022-07-14-08-36-55-924.png
          image-2022-07-14-08-36-55-924.png
          101 kB
        8. image-2022-07-14-08-38-02-566.png
          image-2022-07-14-08-38-02-566.png
          93 kB
        9. image-2022-07-14-12-41-19-603.png
          image-2022-07-14-12-41-19-603.png
          144 kB
        10. image-2022-07-14-12-41-35-251.png
          image-2022-07-14-12-41-35-251.png
          181 kB
        11. requested_files.tar.gz
          337 kB

            Assignee:
            louis.williams@mongodb.com Louis Williams
            Reporter:
            bobthekingofegypt@gmail.com Richard Martin
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: