Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-2169

ConcurrentPool writes to same connection buffer then keeps waiting

    XMLWordPrintableJSON

Details

    • Icon: Task Task
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 3.2.2
    • Connection Management
    • None
    • ubuntu 12

    Description

      This would happen only when most/all writes are unacknowledged.

      Each write happens to last available connection based on code in:
      mongo-java-driver-master/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java:130
      `T t = available.pollLast();`

      Each release of connection again appends back the available connection to end
      /goshposh/mongo-java-driver-master/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java:84
      `available.addLast(t);`

      Situation:
      Trying to use mongo java driver in jruby, seeing lower performance than ruby driver.
      We see that `collection.updateOne` takes about 20 seconds on client but mongo server logs have only 100ms as the time taken for query.
      JavaStack reveals that thread spend most of the time in `com.mongodb.connection.SocketStream.write`

      Logical explanation:
      Assume Mongo is takes 10 seconds to execute a query.
      You have 2 threads and 100 connection pool size.

      Thread 1 writes to 100th connection buffer - fire & forget
      Thread 2 writes to 99th connection buffer - fire & forget
      thread 1 completes writing to buffer ( mongo::unacknowledged), releases back the connection to pool at the last.
      thread 2 completes writing to buffer ( mongo::unacknowledged), releases back the connection to pool at the 100.

      Now thread 1 again makes a write, this time it again picks the last buffer ( originally 99th). This TCP send buffer already had something that had to be picked for mongo, but we wrote into the same buffer since it was available to write.
      Now thread 2 will write to 100th buffer , which is again non-empty.

      Eventually there will be a state where buffer are full and mongo client will just wait on java.net.SocketOutputStream.socketWrite0.

      Although the buffer 1 to 98 are empty it will not be used since connection is available on 99 & 100 ( but the TCP Send buffer is full )

      Trace attached.

      Ruby driver ( we use 1.8.5) doesn't have this problem since it does a random on connection:
      mongo-ruby-driver-1.8.5/lib/mongo/util/pool.rb
      socket = available[rand(available.length)]

      Attachments

        Activity

          People

            Unassigned Unassigned
            gauravshah Gaurav Shah
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: