Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-5418

High number of commands on replica set members in authenticated sharded setup

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.0.4
    • Component/s: Security, Sharding
    • Labels:
      None
    • Environment:
      Ubuntu 10.04, 64bit running on EC2 (replica sets on large instances, config servers and mongos on micro instances)
    • ALL

      We started to experience that our daily jobs which heavily use our MongoDB cluster took really long to finish. Looking at the output of mongostat on our replica sets we saw that the number of commands was much higher than the number of other operations.

      During a typical run of the daily jobs the query count would be around 4000 and the command count below 100 while other operation counts vary between hundreds and a few thousands. At this time the query count was 300-400 and the command count 1200-1600.

      Our mongodb cluster is authenticated and we use the same keyFile for all mongo instances.

      I started to log network communication with mongosniff which reported high number of 'need to login' errors between the replica set members and mongos instances. This has happened before and we could always solve the issue by restarting the mongos instances. I tried to do the same now but unfortunately it didn't help. I also restarted all other pieces of the cluster too but that didn't solve the issue either.

      The only difference between previous occasions and this one is that yesterday I upgraded all nodes from 2.0.2 to 2.0.4.

      Here's a piece of the mongosniff output:

      10.84.214.41:54537  -->> 10.84.214.41:27017 admin.$cmd  66 bytes  id:83b	2107
      	query: { replSetGetStatus: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:54537   79 bytes  id:1d74c	120652 - 2107
      	reply n:1 cursorId: 0
      	{ errmsg: "need to login", ok: 0.0 }
      10.84.214.41:54537  -->> 10.84.214.41:27017 admin.$cmd  58 bytes  id:83c	2108
      	query: { ismaster: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:54537   324 bytes  id:1d74d	120653 - 2108
      	reply n:1 cursorId: 0
      	{ setName: "rs_a", ismaster: true, secondary: false, hosts: [ "mongodb-rsa1:27017", "mongodb-rsa2:27017" ], arbiters: [ "mongodb-c1:27018" ], primary: "mongodb-rsa1:27017", me: "mongodb-rsa1:27017", maxBsonObjectSize: 16777216, ok: 1.0 }
      10.84.214.41:54537  -->> 10.84.214.41:27017 admin.$cmd  66 bytes  id:83d	2109
      	query: { replSetGetStatus: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:54537   79 bytes  id:1d74e	120654 - 2109
      	reply n:1 cursorId: 0
      	{ errmsg: "need to login", ok: 0.0 }
      10.84.214.41:54537  -->> 10.84.214.41:27017 admin.$cmd  58 bytes  id:83e	2110
      	query: { ismaster: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:54537   324 bytes  id:1d74f	120655 - 2110
      	reply n:1 cursorId: 0
      	{ setName: "rs_a", ismaster: true, secondary: false, hosts: [ "mongodb-rsa1:27017", "mongodb-rsa2:27017" ], arbiters: [ "mongodb-c1:27018" ], primary: "mongodb-rsa1:27017", me: "mongodb-rsa1:27017", maxBsonObjectSize: 16777216, ok: 1.0 }
      10.84.214.41:54537  -->> 10.84.214.41:27017 admin.$cmd  66 bytes  id:83f	2111
      	query: { replSetGetStatus: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:54537   79 bytes  id:1d750	120656 - 2111
      	reply n:1 cursorId: 0
      	{ errmsg: "need to login", ok: 0.0 }
      10.244.73.3:44992  -->> 10.244.73.3:27017 admin.$cmd  58 bytes  id:1e426	123942
      	query: { ismaster: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:44992   324 bytes  id:1d751	120657 - 123942
      	reply n:1 cursorId: 0
      	{ setName: "rs_a", ismaster: true, secondary: false, hosts: [ "mongodb-rsa1:27017", "mongodb-rsa2:27017" ], arbiters: [ "mongodb-c1:27018" ], primary: "mongodb-rsa1:27017", me: "mongodb-rsa1:27017", maxBsonObjectSize: 16777216, ok: 1.0 }
      10.244.73.3:44992  -->> 10.244.73.3:27017 admin.$cmd  66 bytes  id:1e427	123943
      	query: { replSetGetStatus: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:44992   79 bytes  id:1d752	120658 - 123943
      	reply n:1 cursorId: 0
      	{ errmsg: "need to login", ok: 0.0 }
      10.244.73.3:44992  -->> 10.244.73.3:27017 admin.$cmd  58 bytes  id:1e428	123944
      	query: { ismaster: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:44992   324 bytes  id:1d753	120659 - 123944
      	reply n:1 cursorId: 0
      	{ setName: "rs_a", ismaster: true, secondary: false, hosts: [ "mongodb-rsa1:27017", "mongodb-rsa2:27017" ], arbiters: [ "mongodb-c1:27018" ], primary: "mongodb-rsa1:27017", me: "mongodb-rsa1:27017", maxBsonObjectSize: 16777216, ok: 1.0 }
      10.244.73.3:44992  -->> 10.244.73.3:27017 admin.$cmd  66 bytes  id:1e429	123945
      	query: { replSetGetStatus: 1 }  ntoreturn: 1 ntoskip: 0
      10.118.63.28:27017  <<--  10.118.63.28:44992   79 bytes  id:1d754	120660 - 123945
      	reply n:1 cursorId: 0
      	{ errmsg: "need to login", ok: 0.0 }
      

            Assignee:
            randolph@mongodb.com Randolph Tan
            Reporter:
            lackac László Bácsi
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: