Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17543

WT secondary fall to “recovery” when MMAP secondary keep going under insert only workload

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Replication, Storage
    • Labels:
    • ALL

      Version - RC11

      Environment:
      3 members replica set with the setting below:

      • Primary WT (compression enabled 1GB cache)
      • Secondary WT no compression (journal and block)
      • Secondary MMAP

      Machine – Dedicated 6 core machine for WT and 4 core for MMAP

       Used hammer.mongo to do insert only workload , 6 threads
       OS: Windows

      MongoDB shell version: 3.0.0-rc11
      connecting to: 127.0.0.1:5002/test
      EitanRs3a:PRIMARY> rs.status()
      {
              "set" : "EitanRs3a",
              "date" : ISODate("2015-03-11T15:23:10.223Z"),
              "myState" : 1,
              "members" : [
                      {
                              "_id" : 0,
                              "name" : "eitan5:5002",
                              "health" : 1,
                              "state" : 1,
                              "stateStr" : "PRIMARY",
                              "uptime" : 10657,
                              "optime" : Timestamp(1426087390, 2870),
                              "optimeDate" : ISODate("2015-03-11T15:23:10Z"),
                              "electionTime" : Timestamp(1426076744, 2),
                              "electionDate" : ISODate("2015-03-11T12:25:44Z"),
                              "configVersion" : 3,
                              "self" : true
                      },
                      {
                              "_id" : 1,
                              "name" : "eitan1:5002",
                              "health" : 1,
                              "state" : 2,
                              "stateStr" : "SECONDARY",
                              "uptime" : 10572,
                              "optime" : Timestamp(1426087297, 6834),
                              "optimeDate" : ISODate("2015-03-11T15:21:37Z"),
                              "lastHeartbeat" : ISODate("2015-03-11T15:23:09.143Z"),
                              "lastHeartbeatRecv" : ISODate("2015-03-11T15:23:10.112Z"
      ),
                              "pingMs" : 0,
                              "syncingTo" : "eitan5:5002",
                              "configVersion" : 3
                      },
                      {
                              "_id" : 2,
                              "name" : "eitan6:5001",
                              "health" : 1,
                              "state" : 3,
                              "stateStr" : "RECOVERING",
                              "uptime" : 10555,
                              "optime" : Timestamp(1426079421, 2354),
                              "optimeDate" : ISODate("2015-03-11T13:10:21Z"),
                              "lastHeartbeat" : ISODate("2015-03-11T15:23:08.363Z"),
                              "lastHeartbeatRecv" : ISODate("2015-03-11T15:23:08.628Z"
      ),
                              "pingMs" : 1,
                              "configVersion" : 3
                      }
              ],
              "ok" : 1
      }
      

      I will work with michael.grundy@10gen.com to understand the why happen before the replica set fall beyond

        1. MMAPSecondery.txt
          273 kB
        2. rs3primary.txt
          4.64 MB
        3. WTSecondery.txt
          195 kB

            Assignee:
            eitan.klein Eitan Klein
            Reporter:
            eitan.klein Eitan Klein
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: