[SERVER-1606] Oplog entries contain repeated fields ($set) Created: 10/Aug/10  Updated: 12/Jul/16  Resolved: 28/Jan/13

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.2.4, 2.4.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Dwight Merriman Assignee: Eliot Horowitz (Inactive)
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-7871 Updates create $set operations in opl... Closed
is duplicated by DOCS-923 Document driver support for duplicate... Closed
is duplicated by SERVER-8605 Oplog contains missing information ? Closed
Related
is related to SERVER-7871 Updates create $set operations in opl... Closed
is related to SERVER-10162 Eliminate repeated $set/$unset in new... Closed
is related to SERVER-718 in JS shell, duplicated fields should... Closed
is related to SERVER-6399 Refactor update() code Closed
Operating System: ALL
Participants:

 Description   

if a change occurs we need to be careful about backward compatibility / mixing of versions with replication, which is common.

Jason Toffaletti to mongodb-user
show details 4:42 AM (4 hours ago)
I started up a mongod v1.4.4 with --master and ran some multi-field
updates like this from mongo shell:

> db.test2.update(

{thing:'stuff'}

, {$set:

{stuff:14}

, $inc:{version:1}})

While doing this I have a pymongo 1.5.1 script running a tailable
cursor on local.oplog.$main, printing out results:

update {u'o2':

{u'_id': ObjectId('4c610b301a632ffb5161c3e8')}

, u'ns':
u'test.test2', u'ts': Timestamp(1281428434, 1), u'o': {u'$set':
{u'version': 2.0}}, u'op': u'u'}
update {u'o2':

{u'_id': ObjectId('4c610b301a632ffb5161c3e8')}

, u'ns':
u'test.test2', u'ts': Timestamp(1281428450, 1), u'o': {u'$set':
{u'version': 3.0}}, u'op': u'u'}
update {u'o2':

{u'_id': ObjectId('4c610b301a632ffb5161c3e8')}

, u'ns':
u'test.test2', u'ts': Timestamp(1281428586, 1), u'o': {u'$set':
{u'version': 4.0}}, u'op': u'u'}

This is strange, the "o" field only shows the update to version. What
does mongo shell show?

> db['oplog.$main'].find(

{op:"u"}

)
{ "ts" :

{ "t" : 1281428434000, "i" : 1 }

, "op" : "u", "ns" :
"test.test2", "o2" :

{ "_id" : ObjectId("4c610b301a632ffb5161c3e8") }

,
"o" : { "$set" :

{ "stuff" : 1 }

, "$set" :

{ "stuff" : 1 }

} }
{ "ts" :

{ "t" : 1281428450000, "i" : 1 }

, "op" : "u", "ns" :
"test.test2", "o2" :

{ "_id" : ObjectId("4c610b301a632ffb5161c3e8") }

,
"o" : { "$set" :

{ "stuff" : 1346 }

, "$set" :

{ "stuff" : 1346 }

} }
{ "ts" :

{ "t" : 1281428586000, "i" : 1 }

, "op" : "u", "ns" :
"test.test2", "o2" :

{ "_id" : ObjectId("4c610b301a632ffb5161c3e8") }

,
"o" : { "$set" :

{ "stuff" : 14 }

, "$set" :

{ "stuff" : 14 }

} }

Wait, what? This doesn't have the version update, but shows the $set
twice. I know mongod replication works, so I must be doing something
wrong?



 Comments   
Comment by RalphSu [ 21/Mar/13 ]

Thanks for quick response , Eliot.

Re-check the v2.4.0 release, confirm fixed. My bad to have v2.2 mongo in path, then try typing mongod instead of ./mongod
to start a v2.4.0 server(which actually a v.2.2.0 server started)....

Many thanks.

Comment by Eliot Horowitz (Inactive) [ 21/Mar/13 ]

Its fixed in 2.4.0, but its fixed in the replication code.
So new entries to the oplog only have 1 $set field.
So any driver that uses a map will work.

Comment by RalphSu [ 21/Mar/13 ]

I'm encounting the same issue on v2.2. As this bug is closed with fix version 2.2.4(don't find the release on download page through) and v2.4.0.

I did try the offcial v2.4.0 on my unbuntu, still see the same thing
> using mongo shell to query oplog.rs collection would see some incorrect $set (all the same). An "o" payload looks below which help nothing on log tracing.
"o" : {
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

},
"$set" : {
"AI" :

{ "v" : "app", "t" : ISODate("2013-03-21T04:18:51.099Z"), "c" : null }

}
}
> By mongodump and local database, and use bsondump, i see something like below, which is much better.
"$set":

{ "_b": "main" }

,
"$set":

{ "_i": "4fbb314fc681caf13e283a76" }

,
"$set":

{ "_l": Date(1 363839531099) }

,
"$set":

{ "_pv": -1 }

,

Above two items seems to be exactly the same as https://groups.google.com/forum/?fromgroups=#!topic/mongodb-user/eDPGj9nS6_A mentioned.

My question is:
1. Is this issued fixed v2.4.0 release as this jira case stated?
2. If the answer to item1 yes, what kind of the fix is taken? Server-side merge the dup-key "$set" in oplog. Or users need to get a better driver that support dup-key to able to read from oplog.rs?
3. Is there a plan to merge the dup-key in oplog in server side for future release?

Comment by auto [ 21/Mar/13 ]

Author:

{u'date': u'2013-01-28T15:55:04Z', u'name': u'Eliot Horowitz', u'email': u'eliot@10gen.com'}

Message: SERVER-1606: don't duplicate update names

Conflicts:

src/mongo/dbtests/updatetests.cpp
Branch: v2.2
https://github.com/mongodb/mongo/commit/679219a64948a9702c639d2a3bdbd20b6be68723

Comment by auto [ 28/Jan/13 ]

Author:

{u'date': u'2013-01-28T15:55:04Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

Message: SERVER-1606: don't duplicate update names
Branch: master
https://github.com/mongodb/mongo/commit/07861f5341d4a8e95cc23037b6f245ca8b9110b5

Comment by Scott Hernandez (Inactive) [ 27/Dec/12 ]

This is becoming more of a critical issues since 2.2.1 where there were other replication changes which seems to have increased the likelihood of this occurring.

Comment by Vasily [ 05/Sep/12 ]

The better way to resolve this problem is combine all $set operations to one.

For example:

{$set:

{stuff:14}

, $inc:{version:1}}

Will be looks like {$set:

{version:1}

, $set:{version:1}} in oplog.

But it should looks like {$set:{stuff:14, version:1}}

Will you fix this bug in closest time?

Comment by Andy Schwerin [ 08/May/12 ]

Scott requests backport to 2.0.6.

Comment by Chip Salzenberg [ 03/Aug/11 ]

So ... any thoughts/progress on this?

Comment by Jason Toffaletti [ 10/Aug/10 ]

Ahh, I think I understand what is going on now. The client libraries are using map data-types that don't properly support different elements having the same key ($set).

Two solutions come to mind:

1) All clients are modified to support STL multimap-like semantics for BSON/JSON Object type. This seems like it would break a lot of client code that is expecting:

o["this"] = 1
o["this"] = 2

to result in

{"this":2}

not

{"this":1, "this":2}

.

An alternative would be to introduce a new multimap type. That is returned if the BSON contains two matching keys, and can be used to create this type of BSON.

2) The format of the oplog can change, for example something like this might work:

{"o":[{_id: ObjectId("4c610b301a632ffb5161c3e8")}, {$set: { "stuff" : 1346 }}, {$set: {"version":3.0}}], ....}

This could potentially break backwards compatibility for newer mongod versions being replicated to older slave versions.

Generated at Thu Feb 08 02:57:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.