Uploaded image for project: 'C Driver'
  1. C Driver
  2. CDRIVER-421

Replica set query stuck after mongo update to 2.6.4

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • Mongo Db shard set up running on version 2.6.4

    Description

      HI,
      I recently updated Mongo version to 2.6.4 from 2.4.4.
      While updating one of my replica secondary crashed with exception and it took almost 6 days to come online.
      Now all systems are running on version 2.6.4.
      My DB performance became slow after that.
      When i check with the currentOp operation on one of the Mongos Servers and found some queries which were running for days.
      The operation is this:
      "opid" : "secondset:175744",
      "active" : true,
      "secs_running" : 1124884,
      "microsecs_running" : NumberLong("1124884403215"),
      "op" : "none",
      "ns" : "",
      "query" : {
      },
      "desc" : "repl writer worker 8",
      "threadId" : "0x7f7bb445d700",
      "waitingForLock" : false,
      "killPending" : true,
      "numYields" : 0,
      "lockStats" :

      { "timeLockedMicros" : { "r" : NumberLong(0), "w" : NumberLong(135) }

      ,
      "timeAcquiringMicros" : {
      }
      }
      There are many processes like this which are showing me this behavior.
      My current set UP has:
      5 Mongos Servers
      3 Config Servers
      2 replica set.
      Error which caused the replica secondary crash was this:
      2014-09-05T05:24:29.211+0530 [rsSync] replSet initial sync data copy, starting syncup
      2014-09-05T05:24:29.212+0530 [rsSync] oplog sync 1 of 3
      2014-09-05T05:24:29.795+0530 [rsSync] replication oplog stream went back in time. previous timestamp: 5407e88e:3 newest timestamp: 5406e5e3:42. attempting to sync directly from primary.
      2014-09-05T05:24:29.904+0530 [repl writer worker 1] ERROR: exception: :: caused by :: 56 An empty update path is not valid. on:

      { ts: Timestamp 1409738211000|67, h: -8833753959820233956, v: 2, op: "u", ns: "ibeat20140903.visitor", o2: { _id: ObjectId('54060caebc88dbf98cc57898') }

      , o: { $set: { : 20, AID-41575656: 17, count: 1938, pgAccessTime: 1409738211851 } } }
      2014-09-05T05:24:29.905+0530 [repl writer worker 1] Fatal Assertion 16361
      2014-09-05T05:24:29.905+0530 [repl writer worker 1]
      ***aborting after fassert() failure
      Surprisingly when I search for the record using mongos server I got nothing.
      db.visitor.findOne({_id: ObjectId('54060caebc88dbf98cc57898') });
      null

      Attachments

        Activity

          People

            Unassigned Unassigned
            virendra.agarwal@timesinternet.in Virendra Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: