Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21446

Mergecursors performance issue when handling targeted query results

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Works as Designed
    • Affects Version/s: 2.6.10
    • Fix Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      1. Launch a sharded cluster with 2 shards:

        mlaunch --replicaset --nodes 1 --sharded shard1 shard2
      

      2. Insert the data in the attached file:

        mongo < data.js
      

      3. Shard the colleciton and set a split point to force all of the inserted data to the secondary shard.

        sh.enableSharding('test')
        db.deal.createIndex({'_id.id':1, '_id.version':1})
        sh.shardCollection('test.deal',{'_id.id':1, '_id.version':1})
        sh.splitAt('test.deal',{'_id.id':100, '_id.version': 100})
      

      4. sh.status should show there are 2 chunks and all chunks with '_id.id: 1' are not on the primary shard.

      {  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }
        test.deal
          shard key: { "_id.id" : 1, "_id.version" : 1 }
          chunks:
            shard2	1
            shard1	1
          { "_id.id" : { "$minKey" : 1 }, "_id.version" : { "$minKey" : 1 } } -->> { "_id.id" : 100, "_id.version" : 100 } on : shard2 Timestamp(2, 0)
          { "_id.id" : 100, "_id.version" : 100 } -->> { "_id.id" : { "$maxKey" : 1 }, "_id.version" : { "$maxKey" : 1 } } on : shard1 Timestamp(2, 1)
      

      5. Log into each of the mongoDB shards, and enable the profiler.

        db.setProfilingLevel(2)
      

      6. Run the following aggregation query.

        db.deal.aggregate([{ "$match" : { "_id.id" : 1}},
                        { "$sort" : { "_id.version" : -1}},
                        { "$project" : { "Deal.dealPrivate.versionEffectiveDate" : 1 ,
                                         "_id.id" : 1 ,
                                         "_id.version" : 1 ,
                                         "Deal.LongLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1 ,
                                         "Deal.ShortLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1}}],
                       { "allowDiskUse" : true});
      

      7. On each shard look at the generated profile data. It should look similar to the following.

      Secondary shard:

      {	"op" : "command",
      	"ns" : "test.$cmd",
      	"command" : {
      		"aggregate" : "deal",
      		"pipeline" : [ ...... ]
        }
        ....
        "millis" : 0
      }
      

      Primary shard:

      {
      	"op" : "command",
      	"ns" : "test.$cmd",
      	"command" : {
      		"aggregate" : "deal",
      		"pipeline" : [
      			{
      				"$mergeCursors" : [
      					{
      						"host" : "Steves-MacBook-Pro.local:27019",
      						"id" : NumberLong("49011188702")
      					}
      				]
      			},
            .......
          ]
        },
        ....
        "millis" : 522
      }
      

      Show
      1. Launch a sharded cluster with 2 shards: mlaunch --replicaset --nodes 1 --sharded shard1 shard2 2. Insert the data in the attached file: mongo < data.js 3. Shard the colleciton and set a split point to force all of the inserted data to the secondary shard. sh.enableSharding('test') db.deal.createIndex({'_id.id':1, '_id.version':1}) sh.shardCollection('test.deal',{'_id.id':1, '_id.version':1}) sh.splitAt('test.deal',{'_id.id':100, '_id.version': 100}) 4. sh.status should show there are 2 chunks and all chunks with '_id.id: 1' are not on the primary shard. { "_id" : "test", "partitioned" : true, "primary" : "shard1" } test.deal shard key: { "_id.id" : 1, "_id.version" : 1 } chunks: shard2 1 shard1 1 { "_id.id" : { "$minKey" : 1 }, "_id.version" : { "$minKey" : 1 } } -->> { "_id.id" : 100, "_id.version" : 100 } on : shard2 Timestamp(2, 0) { "_id.id" : 100, "_id.version" : 100 } -->> { "_id.id" : { "$maxKey" : 1 }, "_id.version" : { "$maxKey" : 1 } } on : shard1 Timestamp(2, 1) 5. Log into each of the mongoDB shards, and enable the profiler. db.setProfilingLevel(2) 6. Run the following aggregation query. db.deal.aggregate([{ "$match" : { "_id.id" : 1}}, { "$sort" : { "_id.version" : -1}}, { "$project" : { "Deal.dealPrivate.versionEffectiveDate" : 1 , "_id.id" : 1 , "_id.version" : 1 , "Deal.LongLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1 , "Deal.ShortLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1}}], { "allowDiskUse" : true}); 7. On each shard look at the generated profile data. It should look similar to the following. Secondary shard: { "op" : "command", "ns" : "test.$cmd", "command" : { "aggregate" : "deal", "pipeline" : [ ...... ] } .... "millis" : 0 } Primary shard: { "op" : "command", "ns" : "test.$cmd", "command" : { "aggregate" : "deal", "pipeline" : [ { "$mergeCursors" : [ { "host" : "Steves-MacBook-Pro.local:27019", "id" : NumberLong("49011188702") } ] }, ....... ] }, .... "millis" : 522 }

      Description

      An issue has been found where queries which were targeted to a single shard suffered significant performance degradation relative to running the same query against an unsharded collection. The issue was tracked down to an unnecessary mergecursors operation being performed on the primary shard (SERVER-7656). A patched version of 2.6 was subsequently produced to prevent this unnecessary step from being performed however there is an open question as to why the performance of the mergecursrors operation is so bad when it is essentially a noop.

      In order to assist engineering with debugging this issue, I have identified a series of steps than can be followed in order to reliable reproduce this issue on MongoDB 2.6.10 (see 'Steps to reproduce').

      The SERVER-7656 patch will clearly stop this from happening for queries targeted to a single shard however there is an open question as to why merging pre-sorted results from a single shard is taking as long as it is.

        Attachments

        1. data.js
          3 kB
        2. runTest.js
          6 kB

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            steve.dalby Stephen Dalby
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: