-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
None
-
Affects Version/s: 2.4.9, 2.4.10
-
Component/s: Replication
-
Labels:
-
Environment:EC2
Linux mongo-005.foo.com 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
-
Linux
Our replication slaves are crashing and we need monit in order to start mongo and keep our servers running.
we have 3 nodes and one arbiter in our replication set.
blah:SECONDARY> rs.status()
{
"set" : "blah",
"date" : ISODate("2014-04-08T17:44:54Z"),
"myState" : 2,
"syncingTo" : "ec2-54-217-204-132.eu-west-1.compute.amazonaws.com:27017",
"members" : [
{
"_id" : 9,
"name" : "ec2.compute.amazonaws.com:27017",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 4418,
"lastHeartbeat" : ISODate("2014-04-08T17:44:52Z"),
"lastHeartbeatRecv" : ISODate("2014-04-08T17:44:53Z"),
"pingMs" : 0
},
{
"_id" : 10,
"name" : "ec2.compute.amazonaws.com:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 4416,
"optime" : Timestamp(1396979093, 117),
"optimeDate" : ISODate("2014-04-08T17:44:53Z"),
"lastHeartbeat" : ISODate("2014-04-08T17:44:53Z"),
"lastHeartbeatRecv" : ISODate("2014-04-08T17:44:53Z"),
"pingMs" : 1
},
{
"_id" : 11,
"name" : "ec2.compute.amazonaws.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 4427,
"optime" : Timestamp(1396979094, 42),
"optimeDate" : ISODate("2014-04-08T17:44:54Z"),
"self" : true
},
{
"_id" : 12,
"name" : "ec2.compute.amazonaws.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 4354,
"optime" : Timestamp(1396979094, 30),
"optimeDate" : ISODate("2014-04-08T17:44:54Z"),
"lastHeartbeat" : ISODate("2014-04-08T17:44:54Z"),
"lastHeartbeatRecv" : ISODate("2014-04-08T17:44:53Z"),
"pingMs" : 0,
"syncingTo" : "ec2.eu-west-1.compute.amazonaws.com:27017"
}
],
"ok" : 1
}
Tue Apr 8 15:41:02.882 [repl writer worker 2] ERROR: writer worker caught exception: E11000 duplicate key error index: reporting_events-production.tracking_game_campaigns.$token_1_game_id_1 dup key: { : null, : null } on: { ts: Timestamp 1396971662000|179, h: -5379447976443473461, v: 2, op: "u", ns: "reporting_events-production.tracking_game_campaigns", o2: { _id: ObjectId('531e6a0c365ad5221f00057a') }, o: { $set: { campaign_id: 1214255, expires_at: new Date(1397058062870), updated_at: new Date(1396971662875) } } }
Tue Apr 8 15:41:37.327 [repl writer worker 3] ERROR: writer worker caught exception: E11000 duplicate key error index: reporting_events-production.tracking_game_campaigns.$token_1_game_id_1 dup key: { : null, : null } on: { ts: Timestamp 1396971662000|179, h: -5379447976443473461, v: 2, op: "u", ns: "reporting_events-production.tracking_game_campaigns", o2: { _id: ObjectId('531e6a0c365ad5221f00057a') }, o: { $set: { campaign_id: 1214255, expires_at: new Date(1397058062870), updated_at: new Date(1396971662875) } } }
Tue Apr 8 15:42:13.895 [repl writer worker 4] ERROR: writer worker caught exception: E11000 duplicate key error index: reporting_events-production.tracking_game_campaigns.$token_1_game_id_1 dup key: { : null, : null } on: { ts: Timestamp 1396971662000|179, h: -5379447976443473461, v: 2, op: "u", ns: "reporting_events-production.tracking_game_campaigns", o2: { _id: ObjectId('531e6a0c365ad5221f00057a') }, o: { $set: { campaign_id: 1214255, expires_at: new Date(1397058062870), updated_at: new Date(1396971662875) } } }
Tue Apr 8 16:30:54.943 [repl writer worker 5] ERROR: writer worker caught exception: E11000 duplicate key error index: reporting_events-production.tracking_game_campaigns.$token_1_game_id_1 dup key: { : null, : null } on: { ts: Timestamp 1396974654000|125, h: -6524155580285384310, v: 2, op: "u", ns: "reporting_events-production.tracking_game_campaigns", o2: { _id: ObjectId('531e6b7dc1be4444e2001b58') }, o: { $set: { campaign_id: 1165644, updated_at: new Date(1396974654930) } } }