[SERVER-35856] Initial sync failed again and again due to unique index conflict Created: 28/Jun/18  Updated: 05/Dec/19  Resolved: 07/Nov/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: CenZheng Assignee: Eric Sedor
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod.log     File rsconf.md    
Operating System: ALL
Participants:

 Description   

Hi, 

I encountered a problem when initial sync failed due to unique index conflict, and it can not be recovered, the error messages are as below:

 

2018-06-28T17:53:32.046+0800 I REPL [initandlisten] Replaying stored operations from { ts: Timestamp 1530177067000|2, t: 6 } (exclusive) to { ts: Timestamp 1530177100000|62, t: 7 } (inclusive).
2018-06-28T17:53:32.047+0800 F STORAGE [initandlisten] Unique index cursor seeing multiple records for key { : "423925:333C49:2397ED:e000", : "com.allfootballapp.news" }
2018-06-28T17:53:32.047+0800 I - [initandlisten] Fatal Assertion 28608 at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 974

 

Please help!



 Comments   
Comment by Eric Sedor [ 07/Nov/19 ]

Understood zhcn381, I will close this for now but feel free to comment again or open a new ticket if this reoccurs!

Comment by CenZheng [ 07/Nov/19 ]

Hi, Eric,

Thanks for paying attention to this old issue. As you can seeing from the attached mongod.log, it happened during a secondary initial sync, not a mongorestore attemp. As long time has passed, I am not able to provide more things. I'll let you know if this issue happens again.

Comment by Eric Sedor [ 07/Nov/19 ]

Hi zhcn381,

Is this issue happening during a mongorestore attempt? That would possibly suggest SERVER-12508.

If not, can you let us know if you are still seeing this issue on 3.4.11, and provide a fresh copy of:

  • The output of rs.conf()
  • A list of collections and indexes in your primary
  • The full logs of the node that you're trying to initial sync, from startup until the crash above

Note that we we would still like to see a list of collections and indexes from the Primary, because the attached mongod.log isn't clearly showing what index build is failing. You can upload these files to this secure upload portal.

Gratefully,
Eric

Comment by CenZheng [ 02/Jul/18 ]

Hi Ramon,

I have provided the rs.conf() and the full logs(collection and index info can be found during the initial sync procedure in the log). I think maybe I'm hitting on https://jira.mongodb.org/browse/SERVER-12508.

Thanks.

Comment by Ramon Fernandez Marina [ 28/Jun/18 ]

zhcn381, we'd like to us for some information so we can diagnose this problem further. Can you please provide:

  • The output of rs.conf()
  • A list of collections and indexes in your primary
  • The full logs of the node that you're trying to initial sync, from startup until the crash above

We may also need to look at the oplog of the initial syncing node after it crashes. You can upload this information to JIRA or to a secure portal we've created for this ticket.

Thanks,
Ramón.

Comment by CenZheng [ 28/Jun/18 ]

One important thing i forgot to mention is that due to the replaying stored oplog after restarted logic, it will fail and restart and fail again for endless, I think this time should just ignore the unique index conflict, since the data will be correct after all oplog have been replayed, or maybe even just perform a resync is better.

Generated at Thu Feb 08 04:41:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.