Core Server
  1. Core Server
  2. SERVER-4532

GetLastError on sharded cluster can report incorrect result

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major - P3 Major - P3
    • Resolution: Fixed
    • Affects Version/s: 1.8.3, 2.2.0, 2.2.1
    • Fix Version/s: 2.2.3, 2.3.2
    • Component/s: Write Ops
    • Labels:
    • Environment:
    • Backport:
      Done
    • Backward Breaking:
      No
    • Operating System:
      ALL
    • Bug Type:
      Logical
    • # Replies:
      7
    • Last comment by Customer:
      false

      Description

      Steps:

      1. Insert object into shard (successfully)
      2. Update object by _id (WriteResult writeResult = dbCollection.update(query, update, false, false, WriteConcern.SAFE)
      query looks like: {_id : XYZ}
      update looks like: {$set : {foo : bar, foo2 : bar2}}
      3. Object is updated

      WriteResult:

      { 
         writeResult =   { 
            "serverUsed":"127.0.0.1:27017/myDB", 
            "singleShard":"ReplSet3/ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.compute-1.amazonaws.com:27018", 
      =>    "n":0, 
            "lastOp":5686019109100191745, 
            "connectionId":1524, 
            "err":null, 
            "ok":1.0, 
      =>    "updatedExisting":true, 
            "wtime":0, 
            "writebackGLE": { 
               "singleShard":"ReplSet3/ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.comput1.amazonaws.com:27018", 
               "n":0, 
               "lastOp":5686019109100191745, 
               "connectionId":1524, 
               "err":null, 
               "ok":1.0 
            }, 
            "initialGLEHost":"ReplSet1/ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.compute-1.amazonaws.com:27018,ec2-xxxx.compute1.amazonaws.com:27018" 
         } 
      

      This is very hard to reproduce since it is very intermittent. Maybe somebody has an vague idea about the exact situation when this might occur. One thing to note is that step 1 and 2 are executed very quickly.

      I had created a thread in the forum: http://groups.google.com/group/mongodb-user/browse_thread/thread/d1ca7e977dc7455b/973cab27e21df385?lnk=gst&q=smigfu#973cab27e21df385

      1. noUpdateButN1.js
        2 kB
        Asya Kamsky
      2. noUpdateButN1inAnotherCollection.js
        2 kB
        Asya Kamsky
      3. shTest.js
        2 kB
        Asya Kamsky

        Issue Links

          Activity

          Hide
          Eliot Horowitz
          added a comment -

          This was fixed in 2.0
          As you said - it was very rare - but possible in 1.8

          Show
          Eliot Horowitz
          added a comment - This was fixed in 2.0 As you said - it was very rare - but possible in 1.8
          Hide
          Philipp Marx
          added a comment -

          So is it safe to assume if "updatedExisting" is "true" an update has been performed?

          The reason I am asking is that we want to check whether the update has successfully been applied and currently we are checking the n-value.

          Show
          Philipp Marx
          added a comment - So is it safe to assume if "updatedExisting" is "true" an update has been performed? The reason I am asking is that we want to check whether the update has successfully been applied and currently we are checking the n-value.
          Hide
          agahd
          added a comment - - edited

          Please reopen this issue since we can reproduce it with version 2.2.0.

          The WriteResult obtained by the mongo driver v2.9.3 is as follows:

          { "serverUsed" : "sx176.ipx/x.x.x.x:27018" , 
          "singleShard" : "offerStoreDE2/s127:27018,s131:27018,s136:27018" , 
          "n" : 0 , 
          "lastOp" : { "$ts" : 1352210785 , "$inc" : 182} , 
          "connectionId" : 2409830 , 
          "err" :  null  , 
          "ok" : 1.0 , 
          "writeback" : { "$oid" : "50991961000000000005dd5f"} , 
          "instanceIdent" : "s136:27018" , 
          "updatedExisting" : true , 
          "wtime" : 0 , 
          "writebackGLE" : {
            "singleShard" : "offerStoreDE2/s127:27018,s131:27018,s136:27018" , 
            "n" : 0 , 
            "lastOp" : { "$ts" : 1352210785 , "$inc" : 182} , 
            "connectionId" : 2409830 , 
            "err" :  null  , 
            "ok" : 1.0} , 
          "initialGLEHost" : "offerStoreDE2/s127:27018,s131:27018,s136:27018"}
          

          We are sure that the update was successful because the new values were written into the DB.

          We observed that the error occurs when the balancer it turned on. When we turn the balancer off, the errors continue BUT when we restart the router (mongos) while keeping the balancer off then these errors wont happen again.
          However, we can't keep the balancer off since our shards would be unbalanced.

          Please advice.

          Show
          agahd
          added a comment - - edited Please reopen this issue since we can reproduce it with version 2.2.0. The WriteResult obtained by the mongo driver v2.9.3 is as follows: { "serverUsed" : "sx176.ipx/x.x.x.x:27018" , "singleShard" : "offerStoreDE2/s127:27018,s131:27018,s136:27018" , "n" : 0 , "lastOp" : { "$ts" : 1352210785 , "$inc" : 182} , "connectionId" : 2409830 , "err" : null , "ok" : 1.0 , "writeback" : { "$oid" : "50991961000000000005dd5f"} , "instanceIdent" : "s136:27018" , "updatedExisting" : true , "wtime" : 0 , "writebackGLE" : { "singleShard" : "offerStoreDE2/s127:27018,s131:27018,s136:27018" , "n" : 0 , "lastOp" : { "$ts" : 1352210785 , "$inc" : 182} , "connectionId" : 2409830 , "err" : null , "ok" : 1.0} , "initialGLEHost" : "offerStoreDE2/s127:27018,s131:27018,s136:27018"} We are sure that the update was successful because the new values were written into the DB. We observed that the error occurs when the balancer it turned on. When we turn the balancer off, the errors continue BUT when we restart the router (mongos) while keeping the balancer off then these errors wont happen again. However, we can't keep the balancer off since our shards would be unbalanced. Please advice.
          Hide
          Asya Kamsky
          added a comment -

          SERVER-7885 which I opened seems like a sub-case of this one. Here's a way to create n=0 on a successful update in the shell with two shards and two mongos processes.

          In one mongos process moveChunk({_id: 1001}, "othershard");
          Then immediately after run (against the other mongos):

          $ mongo localhost:57017/test2 --eval 'db.coll.update({_id:1001},{$set:{"a":"999"}});printjson(db.getLastErrorObj())'
          MongoDB shell version: 2.2.1
          connecting to: localhost:57017/test2
          {
             [...]
          	"n" : 0,
          	"err" : null,
          	"ok" : 1
          }
          

          n is returned as 0 and updatedExisting is not set at all.
          In the logs you can see that correct thing is returned from mongod:
          Last thing from WritebackListener:
          [WriteBackListener-localhost:37017] GLE is

          { singleShard: "asya/localhost:27017,localhost:27018,localhost:27019", updatedExisting: true, n: 1, lastOp: Timestamp 1355026437000|1, connectionId: 32370, err: null, ok: 1.0 }
          Show
          Asya Kamsky
          added a comment - SERVER-7885 which I opened seems like a sub-case of this one. Here's a way to create n=0 on a successful update in the shell with two shards and two mongos processes. In one mongos process moveChunk({_id: 1001}, "othershard"); Then immediately after run (against the other mongos): $ mongo localhost:57017/test2 --eval 'db.coll.update({_id:1001},{$set:{"a":"999"}});printjson(db.getLastErrorObj())' MongoDB shell version: 2.2.1 connecting to: localhost:57017/test2 { [...] "n" : 0, "err" : null, "ok" : 1 } n is returned as 0 and updatedExisting is not set at all. In the logs you can see that correct thing is returned from mongod: Last thing from WritebackListener: [WriteBackListener-localhost:37017] GLE is { singleShard: "asya/localhost:27017,localhost:27018,localhost:27019", updatedExisting: true, n: 1, lastOp: Timestamp 1355026437000|1, connectionId: 32370, err: null, ok: 1.0 }
          Hide
          Asya Kamsky
          added a comment - - edited

          Okay just ran the same test with 2.3.2-pre (nightly from today) and it also fails.

          Ignore previous comment saying that it's been fixed...

          Show
          Asya Kamsky
          added a comment - - edited Okay just ran the same test with 2.3.2-pre (nightly from today) and it also fails. Ignore previous comment saying that it's been fixed...
          Hide
          auto
          added a comment -

          Author:

          {u'date': u'2012-12-12T05:18:43Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

          Message: SERVER-4532 can't call ClientInfo::addShard on things you don't really use
          Branch: master
          https://github.com/mongodb/mongo/commit/587fde4994df19665404160ff1e399dd5af9d6b0

          Show
          auto
          added a comment - Author: {u'date': u'2012-12-12T05:18:43Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'} Message: SERVER-4532 can't call ClientInfo::addShard on things you don't really use Branch: master https://github.com/mongodb/mongo/commit/587fde4994df19665404160ff1e399dd5af9d6b0
          Hide
          auto
          added a comment -

          Author:

          {u'date': u'2012-12-12T05:18:43Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'}

          Message: SERVER-4532 can't call ClientInfo::addShard on things you don't really use

          Conflicts:

          src/mongo/s/s_only.cpp
          Branch: v2.2
          https://github.com/mongodb/mongo/commit/d03939e182aec740fb83a65400ee02ce572752ff

          Show
          auto
          added a comment - Author: {u'date': u'2012-12-12T05:18:43Z', u'email': u'eliot@10gen.com', u'name': u'Eliot Horowitz'} Message: SERVER-4532 can't call ClientInfo::addShard on things you don't really use Conflicts: src/mongo/s/s_only.cpp Branch: v2.2 https://github.com/mongodb/mongo/commit/d03939e182aec740fb83a65400ee02ce572752ff

            People

            • Votes:
              3 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 18 weeks, 5 days ago
                Date of 1st Reply: