Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3278

Master stopped allowing connections, didn't fail over "DR102 too much data written uncommitted"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 1.8.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL

      Description

      Thu Jun 16 16:43:34 [conn35976] core.Core_Model_User Assertion failure ! "DR102 too much data written uncommitted" db/dur_commitjob.cpp 204
      0x51fe32 0x5305cf 0x5e6a28 0x5da88a 0x6947c8 0x698477 0x6948c2 0x60fb65 0x61193f 0x6cb29d 0x6d1520 0x6d23c2 0x548582 0x5cc648 0x35c3855112 0x35c385738f 0x35c38557b3 0x35c384f794 0x35c3855112 0x35c385738f
      /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xb2) [0x51fe32]
      /usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0xbf) [0x5305cf]
      /usr/bin/mongod(_ZN5mongo3dur9CommitJob4noteEPvi+0x198) [0x5e6a28]
      /usr/bin/mongod(_ZN5mongo3dur11DurableImpl10writingPtrEPvj+0xa) [0x5da88a]
      /usr/bin/mongod(ZN5mongo12append_O_ObjEPcRKNS_7BSONObjES3+0x48) [0x6947c8]
      /usr/bin/mongod [0x698477]
      /usr/bin/mongod(_ZN5mongo5logOpEPKcS1_RKNS_7BSONObjEPS2_Pb+0x42) [0x6948c2]
      /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugEPNS_11RemoveSaverE+0x1f05) [0x60fb65]
      /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES2_bbbRNS_7OpDebugE+0x12f) [0x61193f]
      /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4dd) [0x6cb29d]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x15a0) [0x6d1520]
      /usr/bin/mongod(_ZN5mongo14DBDirectClient3sayERNS_7MessageE+0x62) [0x6d23c2]
      /usr/bin/mongod(_ZN5mongo12DBClientBase6updateERKSsNS_5QueryENS_7BSONObjEbb+0x242) [0x548582]
      /usr/bin/mongod(ZN5mongo12mongo_updateEP9JSContextP8JSObjectjPlS4+0x1c8) [0x5cc648]
      /usr/lib64/libjs.so.1(js_Invoke+0x492) [0x35c3855112]
      /usr/lib64/libjs.so.1(js_Interpret+0x14df) [0x35c385738f]
      /usr/lib64/libjs.so.1(js_Invoke+0xb33) [0x35c38557b3]
      /usr/lib64/libjs.so.1 [0x35c384f794]
      /usr/lib64/libjs.so.1(js_Invoke+0x492) [0x35c3855112]
      /usr/lib64/libjs.so.1(js_Interpret+0x14df) [0x35c385738f]
      Thu Jun 16 16:43:34 [conn35976] local.oplog.rs Assertion failure ! "DR102 too much data written uncommitted" db/dur_commitjob.cpp 204

      This message was repeating in the logs (unfortunately we failed to save the logs prior to restart so we don't have any more info from them.

      Trying to connect to the master failed:

      PROD root@docbase1-10-125-50-69 ~ $ mongo localhost:27018
      MongoDB shell version: 1.8.1
      connecting to: localhost:27018/test
      Thu Jun 16 16:39:05 MessagingPort recv() errno:104 Connection reset by peer 127.0.0.1:27018
      Thu Jun 16 16:39:05 SocketException: remote: error: 9001 socket exception [1]
      Thu Jun 16 16:39:05 DBClientCursor::init call() failed
      exception: DBClientBase::findOne: transport error: localhost:27018

      But it was still sending heartbeats

      query:

      { whatsmyuri: 1 }

      "members" : [
      {
      "_id" : 0,
      "name" : "ec2-50-16-30-183.compute-1.amazonaws.com:27018",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 35043,
      "optime" :

      { "t" : 1308256847000, "i" : 28 }

      ,
      "optimeDate" : ISODate("2011-06-16T20:40:47Z"),
      "lastHeartbeat" : ISODate("2011-06-16T20:40:47Z")
      },
      {
      "_id" : 1,
      "name" : "ec2-184-72-213-192.compute-1.amazonaws.com:27018",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "optime" :

      { "t" : 1308254621000, "i" : 7 }

      ,
      "optimeDate" : ISODate("2011-06-16T20:03:41Z"),
      "self" : true
      },

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              redbeard0531 Mathias Stearn
              Reporter:
              raizyr Chris McNabb
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: