Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-49000

number of logical sessions is constantly increasing, including for CloudManager agent

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 3.6.17, 4.0.18
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL

      Description

      We are running a mongodb v3.6 replSet for several months (or even years) already without any problems so far. However, recently our database client, which is using mongodb java driver v3.11 for a decent time already, got the following error:

      ERROR 2020-06-07 18:31:11,705 [ttp-nio-8080-exec-13] Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.mongodb.MongoQueryException: Query failed
       with error code 261 and error message 'cannot add session into the cache' on server mongo-opensearch-03.db00.pro06.eu.idealo.com:27017] with root cause [DirectJDKLog.java:175]
      com.mongodb.MongoQueryException: Query failed with error code 261 and error message 'cannot add session into the cache' on server mongo-opensearch-03.db00.pro06.eu.idealo.com:27017
              at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735)
      

      It turned out that the default of 1 million logical sessions had been reached. We even could not kill any sesssion:

      opensearch:PRIMARY> db.runCommand( { killAllSessions: [ ] } )
      {
      	"ok" : 0,
      	"errmsg" : "cannot add session into the cache",
      	"code" : 261,
      	"codeName" : "TooManyLogicalSessions"
      }

      We had to restart mongod in order to reset all logical sessions.

      opensearch:SECONDARY> db.aggregate( [ { $listLocalSessions: { allUsers: true } },{$group:{_id:"$user",count:{$sum:1}}} ] )
      { "_id" : "admin@admin", "count" : 1 }
      { "_id" : "mms@admin", "count" : 3 }

      We then upgraded from mongodb v3.6 to 4.0.18 to see if this solves the issue. It did not. After 12 days, the number of logical sessions is still constantly increasing:

      opensearch:SECONDARY> db.aggregate( [ { $listLocalSessions: { allUsers: true } },{$group:{_id:"$user",count:{$sum:1}}} ] )
      { "_id" : "opensearch@admin", "count" : 115 }
      { "_id" : "admin@admin", "count" : 2388 }
      { "_id" : "mms@admin", "count" : 1370 }
      

      I'll attach also a screenshot of our dashboard where you see that the number of logical sessions is constantly increasing.

      The user "opensearch" is our usual database client. The number of logical sessions of 125 is reasonably low.
      The user "mms" is mongodb's CloudManager agent. It's been updated recently to the newest version. However, the number of logical sessions of 1370 is quite high already.

      Although we have been running dozens of mongodb v3.6 replSets in production for several years now, this has never happened before. Since everything is rolled out automatically, and therefore uses identical configuration, we cannot explain the error and therefore suspect a mongodb bug.

        Attachments

          Activity

            People

            Assignee:
            dmitry.agranat Dmitry Agranat
            Reporter:
            kaga kay.agahd
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: