[SERVER-6537] Replicaset stop replication Created: 20/Jul/12  Updated: 16/Nov/21  Resolved: 15/Nov/12

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Raymond Assignee: Shaun Verch
Resolution: Duplicate Votes: 0
Labels: replication
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: Centos 5 x64


Issue Links:
Duplicate
duplicates SERVER-6833 Secondary crashes when replicating a ... Closed
Operating System: ALL
Participants:

 Description   

We get following error in our secondary, this is second time we encounter the problem, we have remove all the data in secondary and let it resync all the data in one week ago and seems fixed. But the problem comes again.

"errmsg" : "syncTail: 10068 invalid operator: $id, syncing: { ts: Timestamp 1342667002000|51, h: -3142433914917806080, op: \"u\", ns: \"boc.paper\", o2: { _id:

{ $id: \"4fe0050d217042a83c010000\" }

}, o: { $set:

{ info.difficulty: [ \"1\", \"2\", \"3\" ] }

} }"
},



 Comments   
Comment by Shaun Verch [ 15/Nov/12 ]

Hi David,

For more updates on this, you can follow SERVER-6833. That describes the underlying issue and the fix for that ticket will also fix the behavior you saw here.

I was able to reproduce it using the php driver given your description, so I'm posting that here:

<?php

$m = new Mongo("localhost:30001",array('replicaSet'=>'testreplset'));
$db = $m->dollarsign;
$collection = $db->dollarsign;
$obj = array( "_id" => array( '$foo' => 1 ) );
$collection->insert($obj, array("safe" => true));

?>

Thanks!

Comment by David Gubler [ 22/Oct/12 ]

Unfortunately no, sorry (logrotate took care of that...)

But I had a close look at the logs when it happened and I don't remember seeing anything out of the ordinary before that assert (it came "out of the blue").

Comment by Shaun Verch [ 18/Oct/12 ]

Hi David,

Thanks for the update. Do you have any logs from right before the secondary triggered this assert?

syncTail: 10068 invalid operator: $oid, syncing: { ts: Timestamp 1348485921000|30, h: -4162345707935058041, op: "i", ns: "doodle.pollCreatedLinkTracking", o: { _id: { $oid: "5059a6cf44aef65722ff7302" }, adminEmailLink: 0.0, copiedLink: 0.0, originalLink:

Comment by David Gubler [ 18/Oct/12 ]

Never mind the segfault. It turns out that the server in question had a faulty disk that produced garbage. I guess that crash was due to corrupted data from the disk. Sorry for that.

Comment by Shaun Verch [ 04/Oct/12 ]

Thank you for the bug report. We're looking into this issue, and will let you know if we need any additional information.

Comment by David Gubler [ 25/Sep/12 ]

We have hit the same issue with 2.0.7. One of the affected secondaries says:

syncTail: 10068 invalid operator: $oid, syncing: { ts: Timestamp 1348485921000|30, h: -4162345707935058041, op: "i", ns: "doodle.pollCreatedLinkTracking", o: { _id:

{ $oid: "5059a6cf44aef65722ff7302" }

, adminEmailLink: 0.0, copiedLink: 0.0, originalLink:

The others just stop replicating.

My co-worker says:

works:
{
'_a':

{"$foo":1}

,
b:0
}

doesn't work (it appears that the driver can execute it on the primary, but it will blow up the secondaries):
{
'_id':

{"$foo":1}

,
b:0
}

We hit this issue while fiddling around with Rockmongo (uses the PHP driver).

This is especially annoying because I found no way to re-sync a secondary from the primary (instead of another secondary). Luckily we create LVM snapshots on the primary, thus (I hope) I can recover a secondary using our backup and later all other secondaries from that one...

Moreover, when I try to stop MongoDB on an affected secondary, it segfaults (but that cound be an unrelated problem):

Tue Sep 25 13:39:14 [conn340605] end connection 188.92.145.82:47449
Logstream::get called in uninitialized state
Tue Sep 25 13:39:15 [conn340427] end connection 188.92.145.81:49001
Logstream::get called in uninitialized state
Tue Sep 25 13:39:15 [conn340429] end connection 188.92.145.81:49027
Logstream::get called in uninitialized state
Tue Sep 25 13:39:16 [conn340284] end connection 188.92.145.81:42240
Logstream::get called in uninitialized state
Tue Sep 25 13:39:24 [rsSync] replSet syncing to: xxx.yyy.com:27017
Tue Sep 25 13:39:24 Invalid access at address: 0

Tue Sep 25 13:39:24 Got signal: 11 (Segmentation fault).

Tue Sep 25 13:39:24 Backtrace:
0xa9609a 0xa9678c 0x7f95660d0ff0 0x5b612f 0x5e08a4 0x5e37f2 0x5b70ec 0x5cd991 0x5c52d3 0x5c5e22 0x5cdd0a 0x5c65bf 0x5c7ff0 0x7a606f 0x7a6b27 0x7a8738 0x823a96 0x825eca 0x827f15 0x827f68
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3aa) [0xa9609a]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) [0xa9678c]
/lib/libpthread.so.0(+0xeff0) [0x7f95660d0ff0]
/usr/bin/mongod(_ZN5mongo15assembleRequestERKSsNS_7BSONObjEiiPKS2_iRNS_7MessageE+0xef) [0x5b612f]
/usr/bin/mongod(_ZN5mongo14DBClientCursor13_assembleInitERNS_7MessageE+0x84) [0x5e08a4]
/usr/bin/mongod(_ZN5mongo14DBClientCursor4initEv+0x92) [0x5e37f2]
/usr/bin/mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x3ac) [0x5b70ec]
/usr/bin/mongod(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x5cd991]
/usr/bin/mongod(_ZN5mongo17DBClientInterface5findNERSt6vectorINS_7BSONObjESaIS2_EERKSsNS_5QueryEiiPKS2_i+0xa3) [0x5c52d3]
/usr/bin/mongod(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x72) [0x5c5e22]
/usr/bin/mongod(_ZN5mongo18DBClientConnection10runCommandERKSsRKNS_7BSONObjERS3_i+0x7a) [0x5cdd0a]
/usr/bin/mongod(_ZN5mongo20DBClientWithCommands4authERKSsS2_S2_RSsb+0xdf) [0x5c65bf]
/usr/bin/mongod(_ZN5mongo18DBClientConnection4authERKSsS2_S2_RSsb+0x250) [0x5c7ff0]
/usr/bin/mongod(_ZN5mongo16replAuthenticateEPNS_12DBClientBaseE+0x3cf) [0x7a606f]
/usr/bin/mongod(_ZN5mongo11OplogReader13commonConnectERKSs+0x277) [0x7a6b27]
/usr/bin/mongod(_ZN5mongo11OplogReader7connectESs+0x18) [0x7a8738]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl15_getOplogReaderERNS_11OplogReaderERSs+0x46) [0x823a96]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl8syncTailEv+0x11a) [0x825eca]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xc5) [0x827f15]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x48) [0x827f68]

Generated at Thu Feb 08 03:12:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.