Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22933

Update last opTime to latest after applyOps no-op

    • Fully Compatible
    • ALL
    • Sharding 11 (03/11/16)
    • 0

      applyOps command will return on failures before updating the last optime, which can lead subsequent calls to use a stale optime, and maybe cause other issues.

      For example, applyChunkOps calls applyOps with a pre-condition, and then follows up with a find command to check whether the write succeed (for the case where the first write works but doesn't return a response, and then the retry fails on pre-condition not matching). The find command will have the wrong optime and not find the documents that should be there.

      	
      [js_test:bouncing_count] 2016-03-01T21:28:31.125+0000 c20771| 2016-03-01T21:28:30.343+0000 D COMMAND  [conn64] run command config.$cmd { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 }
      ...
      ...
      [js_test:bouncing_count] 2016-03-01T21:28:31.175+0000 c20771| 2016-03-01T21:28:30.599+0000 I COMMAND  [conn64] command config.chunks command: applyOps { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:430 locks:{ Global: { acquireCount: { r: 6, w: 1, W: 3 } }, Database: { acquireCount: { r: 1, w: 1 } }, Collection: { acquireCount: { r: 1 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_command 256ms
      [js_test:bouncing_count] 2016-03-01T21:28:31.175+0000 c20771| 2016-03-01T21:28:30.599+0000 D NETWORK  [conn64] Socket say send() Bad file descriptor 10.45.44.60:50398
      [js_test:bouncing_count] 2016-03-01T21:28:31.175+0000 c20771| 2016-03-01T21:28:30.599+0000 I NETWORK  [conn64] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.45.44.60:50398]
      ...
      ...
      ...
      ...
      ...
      ...
      [js_test:bouncing_count] 2016-03-01T21:28:36.627+0000 d20764| 2016-03-01T21:28:36.623+0000 D ASIO     [conn7] startCommand: RemoteCommand 78 -- target:ip-10-45-44-60:20770 db:config expDate:2016-03-01T21:29:06.623+0000 cmd:{ applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.633+0000 d20764| 2016-03-01T21:28:36.623+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 78 on host ip-10-45-44-60:20770
      [js_test:bouncing_count] 2016-03-01T21:28:36.635+0000 c20770| 2016-03-01T21:28:36.623+0000 D COMMAND  [conn82] run command config.$cmd { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.636+0000 c20770| 2016-03-01T21:28:36.623+0000 D QUERY    [conn82] Running query: query: { ns: "foo.bar" } sort: { lastmod: -1 } projection: {} ntoreturn=1
      [js_test:bouncing_count] 2016-03-01T21:28:36.637+0000 c20770| 2016-03-01T21:28:36.623+0000 D QUERY    [conn82] score(2.0003) = baseScore(1) + productivity((1 advanced)/(1 works) = 1) + tieBreakers(0.0001 noFetchBonus + 0.0001 noSortBonus + 0.0001 noIxisectBonus = 0.0003)
      [js_test:bouncing_count] 2016-03-01T21:28:36.639+0000 c20770| 2016-03-01T21:28:36.624+0000 I COMMAND  [conn82] query config.chunks query: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } } planSummary: IXSCAN { ns: 1, lastmod: 1 } ntoreturn:1 ntoskip:0 keysExamined:1 docsExamined:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:1 reslen:176 locks:{ Global: { acquireCount: { r: 3, W: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } 0ms
      [js_test:bouncing_count] 2016-03-01T21:28:36.643+0000 c20770| 2016-03-01T21:28:36.624+0000 I COMMAND  [conn82] command config.$cmd command: applyOps { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 }, maxTimeMS: 30000 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:621 locks:{ Global: { acquireCount: { r: 3, W: 1 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_command 0ms
      [js_test:bouncing_count] 2016-03-01T21:28:36.643+0000 d20764| 2016-03-01T21:28:36.624+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Could not time out request 78, operation has already been cleaned up
      [js_test:bouncing_count] 2016-03-01T21:28:36.643+0000 d20764| 2016-03-01T21:28:36.624+0000 W SHARDING [conn7] chunk operation commit failed and metadata will be revalidated :: caused by :: BadValue: pre-condition failed
      [js_test:bouncing_count] 2016-03-01T21:28:36.645+0000 d20764| 2016-03-01T21:28:36.624+0000 D ASIO     [conn7] startCommand: RemoteCommand 80 -- target:ip-10-45-44-60:20772 db:config expDate:2016-03-01T21:29:06.624+0000 cmd:{ find: "chunks", filter: { lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1456867710000|5, t: 4 } }, limit: 1, maxTimeMS: 30000 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.646+0000 d20764| 2016-03-01T21:28:36.624+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Starting asynchronous command 80 on host ip-10-45-44-60:20772
      [js_test:bouncing_count] 2016-03-01T21:28:36.646+0000 c20772| 2016-03-01T21:28:36.624+0000 D COMMAND  [conn64] run command config.$cmd { find: "chunks", filter: { lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1456867710000|5, t: 4 } }, limit: 1, maxTimeMS: 30000 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.647+0000 c20772| 2016-03-01T21:28:36.624+0000 D QUERY    [conn64] Relevant index 0 is kp: { ns: 1, min: 1 } unique name: 'ns_1_min_1' io: { v: 1, unique: true, key: { ns: 1, min: 1 }, name: "ns_1_min_1", ns: "config.chunks" }
      [js_test:bouncing_count] 2016-03-01T21:28:36.648+0000 c20772| 2016-03-01T21:28:36.624+0000 D QUERY    [conn64] Relevant index 1 is kp: { ns: 1, shard: 1, min: 1 } unique name: 'ns_1_shard_1_min_1' io: { v: 1, unique: true, key: { ns: 1, shard: 1, min: 1 }, name: "ns_1_shard_1_min_1", ns: "config.chunks" }
      [js_test:bouncing_count] 2016-03-01T21:28:36.648+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] Relevant index 2 is kp: { ns: 1, lastmod: 1 } unique name: 'ns_1_lastmod_1' io: { v: 1, unique: true, key: { ns: 1, lastmod: 1 }, name: "ns_1_lastmod_1", ns: "config.chunks" }
      [js_test:bouncing_count] 2016-03-01T21:28:36.648+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] Scoring query plan: IXSCAN { ns: 1, lastmod: 1 } planHitEOF=1
      [js_test:bouncing_count] 2016-03-01T21:28:36.649+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] score(1.0003) = baseScore(1) + productivity((0 advanced)/(1 works) = 0) + tieBreakers(0.0001 noFetchBonus + 0.0001 noSortBonus + 0.0001 noIxisectBonus = 0.0003)
      [js_test:bouncing_count] 2016-03-01T21:28:36.649+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] Scoring query plan: IXSCAN { ns: 1, shard: 1, min: 1 } planHitEOF=0
      [js_test:bouncing_count] 2016-03-01T21:28:36.651+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] score(1.0003) = baseScore(1) + productivity((0 advanced)/(1 works) = 0) + tieBreakers(0.0001 noFetchBonus + 0.0001 noSortBonus + 0.0001 noIxisectBonus = 0.0003)
      [js_test:bouncing_count] 2016-03-01T21:28:36.651+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] Scoring query plan: IXSCAN { ns: 1, min: 1 } planHitEOF=0
      [js_test:bouncing_count] 2016-03-01T21:28:36.651+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] score(1.0003) = baseScore(1) + productivity((0 advanced)/(1 works) = 0) + tieBreakers(0.0001 noFetchBonus + 0.0001 noSortBonus + 0.0001 noIxisectBonus = 0.0003)
      [js_test:bouncing_count] 2016-03-01T21:28:36.652+0000 c20772| 2016-03-01T21:28:36.625+0000 D QUERY    [conn64] Winning plan: IXSCAN { ns: 1, lastmod: 1 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.654+0000 c20772| 2016-03-01T21:28:36.625+0000 I COMMAND  [conn64] command config.chunks command: find { find: "chunks", filter: { lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar" }, readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1456867710000|5, t: 4 } }, limit: 1, maxTimeMS: 30000 } planSummary: IXSCAN { ns: 1, lastmod: 1 } keysExamined:0 docsExamined:0 fromMultiPlanner:1 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:0 nreturned:0 reslen:370 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_command 0ms
      [js_test:bouncing_count] 2016-03-01T21:28:36.654+0000 d20764| 2016-03-01T21:28:36.625+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] Could not time out request 80, operation has already been cleaned up
      [js_test:bouncing_count] 2016-03-01T21:28:36.659+0000 d20764| 2016-03-01T21:28:36.625+0000 I -        [conn7] Fatal assertion 34431 BadValue: chunk operation commit failed: version 16|1||56d6095fecb72102510308d7 doesn't exist in namespacefoo.bar. Unable to save chunk ops. Command: { applyOps: [ { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_4.0", lastmod: Timestamp 16000|0, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 4.0 }, max: { _id: 5.0 }, shard: "shard0005" }, o2: { _id: "foo.bar-_id_4.0" } }, { op: "u", b: false, ns: "config.chunks", o: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, o2: { _id: "foo.bar-_id_3.0" } } ], preCondition: [ { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } } ], writeConcern: { w: "majority", wtimeout: 15000 } }. Result: { got: { _id: "foo.bar-_id_3.0", lastmod: Timestamp 16000|1, lastmodEpoch: ObjectId('56d6095fecb72102510308d7'), ns: "foo.bar", min: { _id: 3.0 }, max: { _id: 4.0 }, shard: "shard0004" }, whatFailed: { ns: "config.chunks", q: { query: { ns: "foo.bar" }, orderby: { lastmod: -1 } }, res: { lastmod: Timestamp 15000|1 } }, ok: 0.0, errmsg: "pre-condition failed", code: 2 }
      [js_test:bouncing_count] 2016-03-01T21:28:36.659+0000 d20764| 2016-03-01T21:28:36.625+0000 I -        [conn7]
      [js_test:bouncing_count] 2016-03-01T21:28:36.660+0000 d20764|
      [js_test:bouncing_count] 2016-03-01T21:28:36.660+0000 d20764| ***aborting after fassert() failure
      [js_test:bouncing_count] 2016-03-01T21:28:36.660+0000 d20764|
      [js_test:bouncing_count] 2016-03-01T21:28:36.660+0000 d20764|
      [js_test:bouncing_count] 2016-03-01T21:28:36.660+0000 d20764| 2016-03-01T21:28:36.633+0000 F -        [conn7] Got signal: 6 (Aborted).
      [js_test:bouncing_count] 2016-03-01T21:28:36.661+0000 d20764|
      [js_test:bouncing_count] 2016-03-01T21:28:36.662+0000 d20764|  0x1395f22 0x1394e59 0x1395662 0x7fd71091b790 0x7fd7105aa625 0x7fd7105abe05 0x131ea73 0xfeff9d 0xff9981 0xba7d53 0xba8bc2 0xb01740 0xcbafb5 0x98e34c 0x1341c4d 0x7fd710913a51 0x7fd71066093d
      [js_test:bouncing_count] 2016-03-01T21:28:36.662+0000 d20764| ----- BEGIN BACKTRACE -----
      [js_test:bouncing_count] 2016-03-01T21:28:36.681+0000 d20764| {"backtrace":[{"b":"400000","o":"F95F22","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F94E59"},{"b":"400000","o":"F95662"},{"b":"7FD71090C000","o":"F790"},{"b":"7FD710578000","o":"32625","s":"gsignal"},{"b":"7FD710578000","o":"33E05","s":"abort"},{"b":"400000","o":"F1EA73","s":"_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE"},{"b":"400000","o":"BEFF9D","s":"_ZN5mongo23ChunkMoveOperationState15commitMigrationERKNS_18MigrationSessionIdE"},{"b":"400000","o":"BF9981"},{"b":"400000","o":"7A7D53","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"400000","o":"7A8BC2","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"400000","o":"701740","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"400000","o":"8BAFB5","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"58E34C","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"F41C4D","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7FD71090C000","o":"7A51"},{"b":"7FD710578000","o":"E893D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.3.2-84-g76cc3b3", "gitVersion" : "76cc3b37b1912c35c80084b859efcdc93b104637", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-220.el6.x86_64", "version" : "#1 SMP Wed Nov 9 08:03:13 EST 2011", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0B7D9200639E740D463E20E1D5761B4DF35DDF5A" }, { "b" : "7FFF7C828000", "elfType" : 3, "buildId" : "08F634A1D22DEFF00461D50A7699DACDC97657BF" }, { "b" : "7FD712B7C000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "E0AEE889D5BF1373F2F9EE0D448DBF3F5B5113F0" }, { "b" : "7FD712938000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0C72521270790A1BD52C8F6B989EEA5A575085BF" }, { "b" : "7FD7126EC000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "E4E49DE2554F02ACF2728D1748874101B0709B3A" }, { "b" : "7FD7124C6000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "17A35AEE324676929C7A5C8B4CE54443ED10AC07" }, { "b" : "7FD711FFE000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "78A49421FA60389F8C774BE68F5EF17DF2BD9CE3" }, { "b" : "7FD711D24000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "4CB6272BCAC2270393F559F67E8ED321690F79D5" }, { "b" : "7FD711AB8000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "24A4E7055CABC29C2DDAC730D635A20C08687AC6" }, { "b" : "7FD7116D5000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "42ED5C4138355C8160DE05CBE3E149CC8E256989" }, { "b" : "7FD7114CD000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "58C5A5FF5C82D7BE3113BE36DD87C7004E3C4DB1" }, { "b" : "7FD7112C9000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "B5AE05CEDC0CE917F50A3A468CFA2ACD8592E8F6" }, { "b" : "7FD710FC3000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "28AF9321EBEA9D172CA43E11A60E02D0F7014870" }, { "b" : "7FD710D3F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "989FE3A42CA8CEBDCC185A743896F23A0CF537ED" }, { "b" : "7FD710B29000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "2AC15B051D1B8B53937E3341EA931D0E96F745D9" }, { "b" : "7FD71090C000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D467973C46E563CDCF64B5F12B2D6A50C7A25BA1" }, { "b" : "7FD710578000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A6D15926E61580E250ED91F84FF7517F3970CD83" }, { "b" : "7FD712D96000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "04202A4A8BE624D2193E812A25589E2DD02D5B5C" }, { "b" : "7FD71035E000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F704FA7D21D05EF31E90FB4890FCA7F3D91DA138" }, { "b" : "7FD710127000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "128802B73016BE233837EA9F2DCBC2153ACC2D6A" }, { "b" : "7FD70FE40000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "DC11D5D89BDC77FF242481122D51E5A08DB60DA8" }, { "b" : "7FD70FC14000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "15782495E3AF093E67DDAE9A86436FFC6B3CC4D3" }, { "b" : "7FD70FA10000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "13FFCD68952B7715DDF34C9321D82E3041EA9006" }, { "b" : "7FD70F805000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "44A3A1C1891B4C8170C3DB80E7117A022E5EECD0" }, { "b" : "7FD70F602000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7FD70F3F7000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "8C0C7CAB7F028E4592A8581EB2122FBECAB26B97" }, { "b" : "7FD70F08C000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "545478030DF991A635CC5E3258A3F5D8A7E94561" }, { "b" : "7FD70EE73000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "CAD1498B2AA3531958C579F5CB39D8D6BFB5675B" }, { "b" : "7FD70EC70000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "565D9CDC6BD59EFE0156BAFE21033BE070F014DA" }, { "b" : "7FD70EA05000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "0B73153AA2E650B19153B7E8A57F9C7A965072CD" }, { "b" : "7FD70E7D6000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "7D821C87BEF03F9D7BBFE7FEE591EC5929D1C22C" }, { "b" : "7FD70E5CD000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "E7B49911F1136073DD7DC58E8118CD9A4FBE2A19" }, { "b" : "7FD70E3B7000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7FD70E1A7000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "6855E5BF5B3634C15F01B1043BD892D727EE3C08" }, { "b" : "7FD70DFA4000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "58BAC04A1DB3964A8F594EFFBE4838AD01214EDC" }, { "b" : "7FD70DD85000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "2D0F26E648D9661ABD83ED8B4BBE8F2CFA50393B" }, { "b" : "7FD70DA46000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "2AA334714B9242998869C67DB01BF20A119B0AB7" }, { "b" : "7FD70D835000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, { "b" : "7FD70D61F000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "1C2B39A5003E9DA8FD9C55972C06245E731E6546" }, { "b" : "7FD70D3FE000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "6FF9BAEEEE9DDEEF2DFA5CBD36147A75891C0AD4" }, { "b" : "7FD70D1D1000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "6BDB4E1990D6EBA12A5C8D39A7650DB8798BF568" }, { "b" : "7FD70CFCD000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "A436538388F1F25113FDA834CA2EED524EFA17D6" }, { "b" : "7FD70CDC5000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "26CC708AC7C0FC1797A2340C024F0ADD0CE054D8" }, { "b" : "7FD70CA51000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "54DB4E3C4EC743FE95DD31C9D312E2898724577E" }, { "b" : "7FD70C825000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "3D18834CC92D576DCB1CD0F44BA62D3BFFFD52B7" }, { "b" : "7FD70C620000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "C53F8B39797A277F40F582D8D11D3C2FFF7E5D1E" }, { "b" : "7FD70C41C000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "97F07716D324E086D43CC4D05873E1A16E020468" }, { "b" : "7FD70C1DE000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "7CD7DD1B6C294C61F494519CE3E0D7E114DFB36D" }, { "b" : "7FD70BFD9000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" }, { "b" : "7FD70BDD3000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "4514552B5354286A143770420B38F2D5985D7FA1" }, { "b" : "7FD70BBCE000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "EEAA33A75735D35F4BF25C3C2830B8C90ABDD8B5" }, { "b" : "7FD70B9C1000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "34D8E3E2565DEF4A685D6976831B0372AD456993" }, { "b" : "7FD70B7BB000", "path" : "/usr/lib64/sasl2/libcrammd5.so", "elfType" : 3, "buildId" : "4CC7E695963F5C8B772EDFF456DB67F89E58FBD6" }, { "b" : "7FD70B5B6000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "F8DDC7A3CA1CE5B75719AE0DC821647B609D17B6" }, { "b" : "7FD70B3B1000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "9D19F93E342AA4EE2D646E64642625F365056E5C" }, { "b" : "7FD70B1A9000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "F7BCE9C6BFF4EAF0CB3142B299CF22D094CE4F04" } ] }}
      [js_test:bouncing_count] 2016-03-01T21:28:36.682+0000 d20764|  mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1395f22]
      [js_test:bouncing_count] 2016-03-01T21:28:36.682+0000 d20764|  mongod(+0xF94E59) [0x1394e59]
      [js_test:bouncing_count] 2016-03-01T21:28:36.682+0000 d20764|  mongod(+0xF95662) [0x1395662]
      [js_test:bouncing_count] 2016-03-01T21:28:36.683+0000 d20764|  libpthread.so.0(+0xF790) [0x7fd71091b790]
      [js_test:bouncing_count] 2016-03-01T21:28:36.683+0000 d20764|  libc.so.6(gsignal+0x35) [0x7fd7105aa625]
      [js_test:bouncing_count] 2016-03-01T21:28:36.683+0000 d20764|  libc.so.6(abort+0x175) [0x7fd7105abe05]
      [js_test:bouncing_count] 2016-03-01T21:28:36.683+0000 d20764|  mongod(_ZN5mongo23fassertFailedWithStatusEiRKNS_6StatusE+0x93) [0x131ea73]
      [js_test:bouncing_count] 2016-03-01T21:28:36.684+0000 d20764|  mongod(_ZN5mongo23ChunkMoveOperationState15commitMigrationERKNS_18MigrationSessionIdE+0x20DD) [0xfeff9d]
      [js_test:bouncing_count] 2016-03-01T21:28:36.684+0000 d20764|  mongod(+0xBF9981) [0xff9981]
      [js_test:bouncing_count] 2016-03-01T21:28:36.684+0000 d20764|  mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x463) [0xba7d53]
      [js_test:bouncing_count] 2016-03-01T21:28:36.685+0000 d20764|  mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0x402) [0xba8bc2]
      [js_test:bouncing_count] 2016-03-01T21:28:36.685+0000 d20764|  mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x1F0) [0xb01740]
      [js_test:bouncing_count] 2016-03-01T21:28:36.685+0000 d20764|  mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB35) [0xcbafb5]
      [js_test:bouncing_count] 2016-03-01T21:28:36.686+0000 d20764|  mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x98e34c]
      [js_test:bouncing_count] 2016-03-01T21:28:36.686+0000 d20764|  mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x26D) [0x1341c4d]
      [js_test:bouncing_count] 2016-03-01T21:28:36.686+0000 d20764|  libpthread.so.0(+0x7A51) [0x7fd710913a51]
      [js_test:bouncing_count] 2016-03-01T21:28:36.686+0000 d20764|  libc.so.6(clone+0x6D) [0x7fd71066093d]
      [js_test:bouncing_count] 2016-03-01T21:28:36.686+0000 d20764| -----  END BACKTRACE  -----
      

        1. bouncing_count_failure_log.txt
          8.68 MB
          Dianna Hohensee

            Assignee:
            dianna.hohensee@mongodb.com Dianna Hohensee (Inactive)
            Reporter:
            dianna.hohensee@mongodb.com Dianna Hohensee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: