[SERVER-9828] MONGO replica set is down due to duplicate key issue on one of the databases Created: 29/May/13  Updated: 10/Dec/14  Resolved: 07/Apr/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Venkat Reddimachu Assignee: Unassigned
Resolution: Incomplete Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux mongodb302p.stag.ch3.s.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27 06:06:45 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
MONGO 2.4.3


Attachments: Text File mongod_rs1.log    
Operating System: Linux
Participants:

 Description   

One of the replica set is down due to following issue and not able to start it and appreciate all your suggestions for it.

Wed May 29 14:58:57 [rsSync] replSet still syncing, not yet to minValid optime 519f99e7:3
Wed May 29 14:58:57 [initandlisten] connection accepted from 10.235.46.86:47796 #49 (46 connections now open)
Wed May 29 14:58:57 [initandlisten] connection accepted from 10.235.79.64:50349 #50 (47 connections now open)
Wed May 29 14:58:58 [initandlisten] connection accepted from 10.235.71.190:26662 #51 (48 connections now open)
Wed May 29 14:58:58 [initandlisten] connection accepted from 10.235.71.190:26663 #52 (49 connections now open)
Wed May 29 14:58:58 [conn52] authenticate db: local

{ authenticate: 1, nonce: "19ab8a9ce924f728", user: "__system", key: "394bc2730622d6df6e5fe8883c3fdf35" }

Wed May 29 14:58:59 [conn52] command admin.$cmd command: { replSetGetStatus: 1, $auth: { local:

{ __system: 2 }

} } ntoreturn:1 keyUpdates:0 reslen:717 1332ms
Wed May 29 14:58:59 [rsSyncNotifier] replset setting oplog notifier to mongodb301p.stag.ch3.s.com:10001
Wed May 29 14:58:59 [repl writer worker 1] replication update of non-mod failed: { ts: Timestamp 1369414119000|3, h: 407750364380487357, op: "u", ns: "sywl_site_qa.customerLocation", o2:

{ _id: ObjectId('51841e09e4b01f3f7f47cc3f') }

, o: { _id: ObjectId('51841e09e4b01f3f7f47cc3f'), _class: "com.shc.ecom.sywlocal.services.customer.entity.UserLocationBean", userId: "cardtest17@gmail.com", locationName: "-1", longitude: "GAkAGgBXAg5X", latitude: "AQMWBAdXCQBU", isDefaultLocation: false, kmartUnitNumber: "3914", kmartLocationDistance: "7.782068777401162", kMartAddress:

{ address1: "156 South Gary Ave", address2: "", city: "Bloomingdale", state: "IL", zipCode: "60108" }

} }
Wed May 29 14:58:59 [repl writer worker 1] Fatal Assertion 16359
0xb07561 0xacc8b3 0x9aba19 0xadab5d 0xb4d3d9 0x35dca07851 0x35dc2e890d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x79) [0x9aba19]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
/usr/bin/mongod() [0xb4d3d9]
/lib64/libpthread.so.0() [0x35dca07851]
/lib64/libc.so.6(clone+0x6d) [0x35dc2e890d]
Wed May 29 14:58:59 [repl writer worker 1]

***aborting after fassert() failure

Wed May 29 14:58:59 Got signal: 6 (Aborted).

Wed May 29 14:58:59 Backtrace:
0xb07561 0x5598c9 0x35dc232920 0x35dc2328a5 0x35dc234085 0xacc8ee 0x9aba19 0xadab5d 0xb4d3d9 0x35dca07851 0x35dc2e890d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
/lib64/libc.so.6() [0x35dc232920]
/lib64/libc.so.6(gsignal+0x35) [0x35dc2328a5]
/lib64/libc.so.6(abort+0x175) [0x35dc234085]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x79) [0x9aba19]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
/usr/bin/mongod() [0xb4d3d9]
/lib64/libpthread.so.0() [0x35dca07851]
/lib64/libc.so.6(clone+0x6d) [0x35dc2e890d]

Please let me know if you need more information on it.

Thanks,
Venkat



 Comments   
Comment by Thomas Rueckstiess [ 07/Apr/14 ]

Hi Venkat,

I've reviewed the high-verbosity logs but couldn't see anything to get to the bottom of the issue. As we haven't heard back from you for a while, and there isn't enough information to diagnose what the problem was, I'll go ahead and close the issue now. If this is re-occurring, please feel free to re-open the ticket with additional information.

Regards,
Thomas

Comment by Daniel Pasette (Inactive) [ 28/Jun/13 ]

Venkat, I apologize for the long delay in response. Is this issue still occurring?

Comment by Daniel Pasette (Inactive) [ 28/Jun/13 ]

Venkat, I apologize for the long delay in response. Is this issue still occurring?

Comment by Venkat Reddimachu [ 04/Jun/13 ]

I have attached log file after enabling verbose logging enabled with --vvvvv. Please let me know if you need more on it.

Thanks,
Venkat

Comment by Daniel Pasette (Inactive) [ 03/Jun/13 ]

Hi Venkat,
Would you be able to restart the downed server with higher verbosity logging --vvvvv and attach the logs?

Generated at Thu Feb 08 03:21:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.