[SERVER-24231] DuplicateKey exeception while replaying the Oplog with unique index Created: 20/May/16  Updated: 04/Aug/20  Resolved: 20/May/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Damien Gasparina Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File replay.sh    
Issue Links:
Duplicate
duplicates TOOLS-176 Dump/Restore with --oplog not point-i... Closed
Operating System: ALL
Steps To Reproduce:

#!/bin/sh
pkill mongod
sleep 5
mkdir db1
rm -rf db1/* db1_backup/ dump/
mongod --dbpath ./db1 --fork --syslog --replSet test
sleep 2
 
# initializing the db
mongo --eval 'rs.initiate()' && sleep 2
mongo --eval 'db.getCollection("test").createIndex({a: 1}, {unique: true})'
mongo --eval 'db.getCollection("test").insert({a: 1})'
mongo --eval 'db.getCollection("test").update({a: 1}, {$set: {a: 2}})'
mongo --eval 'db.getCollection("test").remove({a: 2})'
mongo --eval 'db.getCollection("test").insert({a: 2})'
 
# let's perform a point in time restore using the oplog now
mongodump -d local -c oplog.rs
mv dump/local/oplog.rs.bson dump/oplog.bson
rm -rf dump/local
 
mongorestore --oplogReplay
# failed: restore error: error applying oplog: applyOps: E11000 duplicate key error collection: test.test index: a_1 dup key: { : 2.0 }

Participants:

 Description   

While performing the applyOps command, duplicateKey exception are ignored during insertion but not on update operations.

This could result in a situation where oplog would not be idempotent with a unique index as update could fail where they should not. For example:

db.test.createIndex({a: 1}, {unique: true})
db.test.insert({a: 1})
db.test.update({a: 1}, {$set: {a: 2, b: 1}})
db.test.remove({a: 2})
db.test.insert({a: 2})

if you replay all oplog entries whereas the last insert has already been done ({a: 2} is present in the collection), mongod will fail with:

db.test.update({a: 1}, {$set: {a: 2}})
applyOps: E11000 duplicate key error collection: test.test index: a_1 dup key: { : 2.0 }

This is specially painful while you are performing point in time restore as mongorestore is stopping at the first error encountered.

Cheers,
Damien



 Comments   
Comment by Юрий Соколов [ 04/Aug/20 ]

Now with https://jira.mongodb.org/browse/SERVER-31507 done it is possible to specify applyOps parameter:
`oplogApplicationMode: "InitialSync"` or `oplogApplicationMode: "Recovering"` or `oplogApplicationMode: "Secondary"`.
One of this modes should be suitable for mongorestore.

https://github.com/mongodb/mongo/commit/ced3d3341a2aac6a11297f7dcc2c3c6d2c0e3bec

Comment by Eric Milkie [ 20/May/16 ]

Thanks for filing this; it's a known issue and we are tracking the fix at TOOLS-176 and the related SERVER tickets linked to that one.

Generated at Thu Feb 08 04:05:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.