-
Type: Question
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 2.2.2
-
Component/s: None
-
None
-
Environment:CentOS 6.3
We have several shardings, each sharding has 3 nodes(1 primary, 1 slave, 1 arbiter). On one of those shardings, slave crashes for many times due to the same reason:
========================================================================================================================
Log from slave node:
Mon Jan 14 20:17:16 [rsBackgroundSync] replSet syncing to: test-mongodb-online01.company.internal:27018
Mon Jan 14 20:17:17 [conn361741] end connection 10.10.111.111:7099 (131 connections now open)
Mon Jan 14 20:17:17 [rsSync] replSet error possible failover clock skew issue? 50f3f741:305
Mon Jan 14 20:17:17 [conn361832] end connection 10.10.112.112:13124 (130 connections now open)
Mon Jan 14 20:17:17 [repl writer worker 1] ERROR: writer worker caught exception: E11000 duplicate key error index: test.test.$user_id_1_doc_id_1 dup key: { : "40b2d10559dd731750839a20b78ac075", : 331282 } on: { ts: Timestamp 1358165814000|175, h: -7276577398417103929, v: 2, op: "i", ns: "test.test", o:
}
Mon Jan 14 20:17:17 [repl writer worker 1] Fatal Assertion 16360
0xaffd31 0xac5323 0x9a2e56 0xad35cd 0xb45ba9 0x3d56407851 0x3d560e767d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xaffd31]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xac5323]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156) [0x9a2e56]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xad35cd]
/usr/bin/mongod() [0xb45ba9]
/lib64/libpthread.so.0() [0x3d56407851]
/lib64/libc.so.6(clone+0x6d) [0x3d560e767d]
Mon Jan 14 20:17:17 [repl writer worker 1]
***aborting after fassert() failure
========================================================================================================================
The following shows the indexes of our collection, user_id is our shard key:
test1:PRIMARY> use test
switched to db test
test1:PRIMARY> db.test.getIndexes()
[
{
"v" : 1,
"key" :
,
"ns" : "test.test",
"name" : "id"
},
{
"v" : 1,
"key" :
,
"ns" : "test.test",
"name" : "user_id_1"
},
{
"v" : 1,
"key" :
,
"unique" : true,
"ns" : "test.test",
"name" : "user_id_1_doc_id_1"
}
]
We have examined the oplog regarding the specific key, and we don't find any clues that we break the idempotency. Could anyone give me some advices? Thanks.
test1:PRIMARY> use local
switched to db local
test1:PRIMARY> db.oplog.rs.find(
)
{ "ts" : Timestamp(1358165814000, 175), "h" : NumberLong("-7276577398417103929"), "v" : 2, "op" : "i", "ns" : "test.test", "o" :
}
test1:PRIMARY> db.oplog.rs.find(
{ "op" : "u", "o2._id" : "2013-01-14_40b2d10559dd731750839a20b78ac075_331282" })
{ "ts" : Timestamp(1358165862000, 502), "h" : NumberLong("-2969379546764696617"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165857 }, "$set" :
{ "col4" : 1358165857 }, "$set" :
{ "col4" : 1358165857 }, "$set" :
{ "col4" : 1358165857 }, "$set" :
{ "col4" : 1358165857 } } }
{ "ts" : Timestamp(1358165862000, 847), "h" : NumberLong("-9174841252872684216"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 } } }
{ "ts" : Timestamp(1358165865000, 819), "h" : NumberLong("8728551110388921671"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 } } }
{ "ts" : Timestamp(1358165866000, 149), "h" : NumberLong("-5398085733751542631"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 }, "$set" :
{ "col4" : 1358165860 } } }
{ "ts" : Timestamp(1358165868000, 974), "h" : NumberLong("-472422500396759906"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165862 }, "$set" :
{ "col4" : 1358165862 }, "$set" :
{ "col4" : 1358165862 }, "$set" :
{ "col4" : 1358165862 }, "$set" :
{ "col4" : 1358165862 } } }
{ "ts" : Timestamp(1358165869000, 1635), "h" : NumberLong("-2936801391795544718"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165847 }, "$set" :
{ "col4" : 1358165847 }, "$set" :
{ "col4" : 1358165847 }, "$set" :
{ "col4" : 1358165847 }, "$set" :
{ "col4" : 1358165847 } } }
{ "ts" : Timestamp(1358165871000, 328), "h" : NumberLong("4783438901816552532"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165868 }, "$set" :
{ "col4" : 1358165868 }, "$set" :
{ "col4" : 1358165868 }, "$set" :
{ "col4" : 1358165868 }, "$set" :
{ "col4" : 1358165868 } } }
{ "ts" : Timestamp(1358165871000, 923), "h" : NumberLong("-7701283919644547362"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165831 }, "$set" :
{ "col4" : 1358165831 }, "$set" :
{ "col4" : 1358165831 }, "$set" :
{ "col4" : 1358165831 }, "$set" :
{ "col4" : 1358165831 } } }
{ "ts" : Timestamp(1358165877000, 217), "h" : NumberLong("905044855275590032"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165858 }, "$set" :
{ "col4" : 1358165858 }, "$set" :
{ "col4" : 1358165858 }, "$set" :
{ "col4" : 1358165858 }, "$set" :
{ "col4" : 1358165858 } } }
{ "ts" : Timestamp(1358165878000, 321), "h" : NumberLong("1021697642751326149"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165814 }, "$set" :
{ "col4" : 1358165814 }, "$set" :
{ "col4" : 1358165814 }, "$set" :
{ "col4" : 1358165814 }, "$set" :
{ "col4" : 1358165814 } } }
{ "ts" : Timestamp(1358165878000, 1479), "h" : NumberLong("-1093460978146924040"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165861 }, "$set" :
{ "col4" : 1358165861 }, "$set" :
{ "col4" : 1358165861 }, "$set" :
{ "col4" : 1358165861 }, "$set" :
{ "col4" : 1358165861 } } }
{ "ts" : Timestamp(1358165881000, 253), "h" : NumberLong("7582300884665231609"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 } } }
{ "ts" : Timestamp(1358165886000, 194), "h" : NumberLong("-6977646000941125783"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165841 }, "$set" :
{ "col4" : 1358165841 }, "$set" :
{ "col4" : 1358165841 }, "$set" :
{ "col4" : 1358165841 }, "$set" :
{ "col4" : 1358165841 } } }
{ "ts" : Timestamp(1358165891000, 8), "h" : NumberLong("4878117036357406261"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 } } }
{ "ts" : Timestamp(1358165925000, 1403), "h" : NumberLong("-4218451351552474554"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165892 }, "$set" :
{ "col4" : 1358165892 }, "$set" :
{ "col4" : 1358165892 }, "$set" :
{ "col4" : 1358165892 }, "$set" :
{ "col4" : 1358165892 } } }
{ "ts" : Timestamp(1358165926000, 37), "h" : NumberLong("8166869776502988491"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 } } }
{ "ts" : Timestamp(1358165926000, 1020), "h" : NumberLong("-2018291843739556146"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165913 }, "$set" :
{ "col4" : 1358165913 }, "$set" :
{ "col4" : 1358165913 }, "$set" :
{ "col4" : 1358165913 }, "$set" :
{ "col4" : 1358165913 } } }
{ "ts" : Timestamp(1358165926000, 1578), "h" : NumberLong("-728060886959932591"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165888 }, "$set" :
{ "col4" : 1358165888 }, "$set" :
{ "col4" : 1358165888 }, "$set" :
{ "col4" : 1358165888 }, "$set" :
{ "col4" : 1358165888 } } }
{ "ts" : Timestamp(1358165932000, 363), "h" : NumberLong("-2899178056259116954"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165873 }, "$set" :
{ "col4" : 1358165873 }, "$set" :
{ "col4" : 1358165873 }, "$set" :
{ "col4" : 1358165873 }, "$set" :
{ "col4" : 1358165873 } } }
{ "ts" : Timestamp(1358165933000, 361), "h" : NumberLong("3881140476962009969"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 }, "$set" :
{ "col4" : 1358165883 } } }
{ "ts" : Timestamp(1358165939000, 359), "h" : NumberLong("-7011314542025612956"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 } } }
{ "ts" : Timestamp(1358165942000, 1288), "h" : NumberLong("8584958629205924536"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 }, "$set" :
{ "col4" : 1358165882 } } }
{ "ts" : Timestamp(1358165983000, 778), "h" : NumberLong("-1825276314084065452"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165918 }, "$set" :
{ "col4" : 1358165918 }, "$set" :
{ "col4" : 1358165918 }, "$set" :
{ "col4" : 1358165918 }, "$set" :
{ "col4" : 1358165918 } } }
{ "ts" : Timestamp(1358165991000, 747), "h" : NumberLong("-6136876944333387998"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165935 }, "$set" :
{ "col4" : 1358165935 }, "$set" :
{ "col4" : 1358165935 }, "$set" :
{ "col4" : 1358165935 }, "$set" :
{ "col4" : 1358165935 } } }
{ "ts" : Timestamp(1358165992000, 568), "h" : NumberLong("-628829561541387552"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 } } }
{ "ts" : Timestamp(1358165992000, 1586), "h" : NumberLong("2249883926090089681"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165880 }, "$set" :
{ "col4" : 1358165880 }, "$set" :
{ "col4" : 1358165880 }, "$set" :
{ "col4" : 1358165880 }, "$set" :
{ "col4" : 1358165880 } } }
{ "ts" : Timestamp(1358165993000, 913), "h" : NumberLong("-2406566488359488622"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165986 }, "$set" :
{ "col4" : 1358165986 }, "$set" :
{ "col4" : 1358165986 }, "$set" :
{ "col4" : 1358165986 }, "$set" :
{ "col4" : 1358165986 } } }
{ "ts" : Timestamp(1358166031000, 414), "h" : NumberLong("2038488693111769456"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165985 }, "$set" :
{ "col4" : 1358165985 }, "$set" :
{ "col4" : 1358165985 }, "$set" :
{ "col4" : 1358165985 }, "$set" :
{ "col4" : 1358165985 } } }
{ "ts" : Timestamp(1358166036000, 694), "h" : NumberLong("9076849425697437764"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 }, "$set" :
{ "col4" : 1358165981 } } }
{ "ts" : Timestamp(1358166050000, 1186), "h" : NumberLong("3767029040654546262"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 } } }
{ "ts" : Timestamp(1358166057000, 465), "h" : NumberLong("4054304491520021028"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166056 }, "$set" :
{ "col4" : 1358166056 }, "$set" :
{ "col4" : 1358166056 }, "$set" :
{ "col4" : 1358166056 }, "$set" :
{ "col4" : 1358166056 } } }
{ "ts" : Timestamp(1358166057000, 867), "h" : NumberLong("-8241200638844377915"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166049 }, "$set" :
{ "col4" : 1358166049 }, "$set" :
{ "col4" : 1358166049 }, "$set" :
{ "col4" : 1358166049 }, "$set" :
{ "col4" : 1358166049 } } }
{ "ts" : Timestamp(1358166093000, 283), "h" : NumberLong("8163250488038851407"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 }, "$set" :
{ "col4" : 1358165996 } } }
{ "ts" : Timestamp(1358166114000, 1077), "h" : NumberLong("3933283505264467446"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166073 }, "$set" :
{ "col4" : 1358166073 }, "$set" :
{ "col4" : 1358166073 }, "$set" :
{ "col4" : 1358166073 }, "$set" :
{ "col4" : 1358166073 } } }
{ "ts" : Timestamp(1358166116000, 1092), "h" : NumberLong("-3910445542003046573"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166059 }, "$set" :
{ "col4" : 1358166059 }, "$set" :
{ "col4" : 1358166059 }, "$set" :
{ "col4" : 1358166059 }, "$set" :
{ "col4" : 1358166059 } } }
{ "ts" : Timestamp(1358166123000, 19), "h" : NumberLong("-3379893690547749158"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 }, "$set" :
{ "col4" : 1358165899 } } }
{ "ts" : Timestamp(1358166124000, 222), "h" : NumberLong("-1554362852666524628"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166046 }, "$set" :
{ "col4" : 1358166046 }, "$set" :
{ "col4" : 1358166046 }, "$set" :
{ "col4" : 1358166046 }, "$set" :
{ "col4" : 1358166046 } } }
{ "ts" : Timestamp(1358166124000, 1013), "h" : NumberLong("-6412016679312996442"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166100 }, "$set" :
{ "col4" : 1358166100 }, "$set" :
{ "col4" : 1358166100 }, "$set" :
{ "col4" : 1358166100 }, "$set" :
{ "col4" : 1358166100 } } }
{ "ts" : Timestamp(1358166162000, 1284), "h" : NumberLong("-3760011584546642058"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166058 }, "$set" :
{ "col4" : 1358166058 }, "$set" :
{ "col4" : 1358166058 }, "$set" :
{ "col4" : 1358166058 }, "$set" :
{ "col4" : 1358166058 } } }
{ "ts" : Timestamp(1358166163000, 190), "h" : NumberLong("-1824466010869690902"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165987 }, "$set" :
{ "col4" : 1358165987 }, "$set" :
{ "col4" : 1358165987 }, "$set" :
{ "col4" : 1358165987 }, "$set" :
{ "col4" : 1358165987 } } }
{ "ts" : Timestamp(1358166170000, 450), "h" : NumberLong("-6006134238326624856"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166167 }, "$set" :
{ "col4" : 1358166167 }, "$set" :
{ "col4" : 1358166167 }, "$set" :
{ "col4" : 1358166167 }, "$set" :
{ "col4" : 1358166167 } } }
{ "ts" : Timestamp(1358166173000, 1574), "h" : NumberLong("-38881893422671401"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 } } }
{ "ts" : Timestamp(1358166179000, 141), "h" : NumberLong("-4873708995553826291"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166108 }, "$set" :
{ "col4" : 1358166108 }, "$set" :
{ "col4" : 1358166108 }, "$set" :
{ "col4" : 1358166108 }, "$set" :
{ "col4" : 1358166108 } } }
{ "ts" : Timestamp(1358166187000, 372), "h" : NumberLong("2509842530284094589"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166178 }, "$set" :
{ "col4" : 1358166178 }, "$set" :
{ "col4" : 1358166178 }, "$set" :
{ "col4" : 1358166178 }, "$set" :
{ "col4" : 1358166178 } } }
{ "ts" : Timestamp(1358179966000, 593), "h" : NumberLong("-4290646640946455411"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165806 }, "$set" :
{ "col4" : 1358165806 }, "$set" :
{ "col4" : 1358165806 }, "$set" :
{ "col4" : 1358165806 }, "$set" :
{ "col4" : 1358165806 } } }
{ "ts" : Timestamp(1358179972000, 725), "h" : NumberLong("7234604075717479874"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358165803 }, "$set" :
{ "col4" : 1358165803 }, "$set" :
{ "col4" : 1358165803 }, "$set" :
{ "col4" : 1358165803 }, "$set" :
{ "col4" : 1358165803 } } }
{ "ts" : Timestamp(1358180031000, 1444), "h" : NumberLong("7308405316679414687"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166170 }, "$set" :
{ "col4" : 1358166170 }, "$set" :
{ "col4" : 1358166170 }, "$set" :
{ "col4" : 1358166170 }, "$set" :
{ "col4" : 1358166170 } } }
{ "ts" : Timestamp(1358180102000, 1710), "h" : NumberLong("3814167846358794846"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166102 }, "$set" :
{ "col4" : 1358166102 }, "$set" :
{ "col4" : 1358166102 }, "$set" :
{ "col4" : 1358166102 }, "$set" :
{ "col4" : 1358166102 } } }
{ "ts" : Timestamp(1358180154000, 1605), "h" : NumberLong("-8728984085139165017"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 }, "$set" :
{ "col4" : 1358166096 } } }
{ "ts" : Timestamp(1358180158000, 669), "h" : NumberLong("-4665390581863594269"), "v" : 2, "op" : "u", "ns" : "test.test", "o2" :
, "o" : { "$set" :
{ "col4" : 1358166117 }, "$set" :
{ "col4" : 1358166117 }, "$set" :
{ "col4" : 1358166117 }, "$set" :
{ "col4" : 1358166117 }, "$set" :
{ "col4" : 1358166117 }} }