-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 1.8.1
-
Component/s: Replication
-
None
-
Environment:Linux Kernel 2.6.32-309-ec2, x86_64, EC2 m2.2xlarge with Raid0 created using mdadm on EBS volumes of 1TB each.
-
Linux
I have two secondaries, both of which stopped replication at around the same time with the following messages:
----------------------------------------------------------------------------------------------------------------------------------------------
Secondary 1:
Mon May 2 02:10:39 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
Mon May 2 02:10:50 [replica set sync] replSet syncThread: 13106 nextSafe():
Mon May 2 02:11:02 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
.....
Mon May 2 08:24:56 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
Mon May 2 08:25:07 [replica set sync] replSet error RS102 too stale to catch up, at least from ip-10-84-41-152:27017
---------------------------------------------------------------------------------------------------------------------------------------------
Secondary 2:
Mon May 2 02:10:40 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
Mon May 2 02:10:52 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
Mon May 2 02:11:03 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
....
Mon May 2 05:11:02 [replica set sync] replSet syncThread: 1000 replSet source for syncing doesn't seem to be await capable – is it an older version of mongodb?
....
Mon May 2 08:24:58 [replica set sync] replSet syncThread: 10003 failing update: objects in a capped ns cannot grow
Mon May 2 08:25:09 [ReplSetHealthPollTask] replSet member ip-10-116-71-169 RECOVERING
---------------------------------------------------------------------------------------------------------------------------------------------
I didn't have any alerts setup on these, and this caused the replicas to go stale and I am now getting them back up with a full resync, etc.
I use a lot of capped collections in my code, but I don't update any of them in place. This looks like a MongoDB oplog capped collection issue.
Any help would be appreciated.