Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21446

Mergecursors performance issue when handling targeted query results

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.10
    • Component/s: Sharding
    • Labels:
      None
    • ALL
    • Hide

      1. Launch a sharded cluster with 2 shards:

        mlaunch --replicaset --nodes 1 --sharded shard1 shard2

      2. Insert the data in the attached file:

        mongo < data.js

      3. Shard the colleciton and set a split point to force all of the inserted data to the secondary shard.

        sh.enableSharding('test')
  db.deal.createIndex({'_id.id':1, '_id.version':1})
  sh.shardCollection('test.deal',{'_id.id':1, '_id.version':1})
  sh.splitAt('test.deal',{'_id.id':100, '_id.version': 100})

      4. sh.status should show there are 2 chunks and all chunks with '_id.id: 1' are not on the primary shard.

      {  "_id" : "test",  "partitioned" : true,  "primary" : "shard1" }
  test.deal
    shard key: { "_id.id" : 1, "_id.version" : 1 }
    chunks:
      shard2	1
      shard1	1
    { "_id.id" : { "$minKey" : 1 }, "_id.version" : { "$minKey" : 1 } } -->> { "_id.id" : 100, "_id.version" : 100 } on : shard2 Timestamp(2, 0)
    { "_id.id" : 100, "_id.version" : 100 } -->> { "_id.id" : { "$maxKey" : 1 }, "_id.version" : { "$maxKey" : 1 } } on : shard1 Timestamp(2, 1)

      5. Log into each of the mongoDB shards, and enable the profiler.

        db.setProfilingLevel(2)

      6. Run the following aggregation query.

        db.deal.aggregate([{ "$match" : { "_id.id" : 1}},
                  { "$sort" : { "_id.version" : -1}},
                  { "$project" : { "Deal.dealPrivate.versionEffectiveDate" : 1 ,
                                   "_id.id" : 1 ,
                                   "_id.version" : 1 ,
                                   "Deal.LongLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1 ,
                                   "Deal.ShortLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1}}],
                 { "allowDiskUse" : true});

      7. On each shard look at the generated profile data. It should look similar to the following.

      Secondary shard:

      {	"op" : "command",
	"ns" : "test.$cmd",
	"command" : {
		"aggregate" : "deal",
		"pipeline" : [ ...... ]
  }
  ....
  "millis" : 0
}

      Primary shard:

      {
	"op" : "command",
	"ns" : "test.$cmd",
	"command" : {
		"aggregate" : "deal",
		"pipeline" : [
			{
				"$mergeCursors" : [
					{
						"host" : "Steves-MacBook-Pro.local:27019",
						"id" : NumberLong("49011188702")
					}
				]
			},
      .......
    ]
  },
  ....
  "millis" : 522
}
      Show
      1. Launch a sharded cluster with 2 shards: mlaunch --replicaset --nodes 1 --sharded shard1 shard2 2. Insert the data in the attached file: mongo < data.js 3. Shard the colleciton and set a split point to force all of the inserted data to the secondary shard. sh.enableSharding('test') db.deal.createIndex({'_id.id':1, '_id.version':1}) sh.shardCollection('test.deal',{'_id.id':1, '_id.version':1}) sh.splitAt('test.deal',{'_id.id':100, '_id.version': 100}) 4. sh.status should show there are 2 chunks and all chunks with '_id.id: 1' are not on the primary shard. { "_id" : "test", "partitioned" : true, "primary" : "shard1" } test.deal shard key: { "_id.id" : 1, "_id.version" : 1 } chunks: shard2 1 shard1 1 { "_id.id" : { "$minKey" : 1 }, "_id.version" : { "$minKey" : 1 } } -->> { "_id.id" : 100, "_id.version" : 100 } on : shard2 Timestamp(2, 0) { "_id.id" : 100, "_id.version" : 100 } -->> { "_id.id" : { "$maxKey" : 1 }, "_id.version" : { "$maxKey" : 1 } } on : shard1 Timestamp(2, 1) 5. Log into each of the mongoDB shards, and enable the profiler. db.setProfilingLevel(2) 6. Run the following aggregation query. db.deal.aggregate([{ "$match" : { "_id.id" : 1}}, { "$sort" : { "_id.version" : -1}}, { "$project" : { "Deal.dealPrivate.versionEffectiveDate" : 1 , "_id.id" : 1 , "_id.version" : 1 , "Deal.LongLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1 , "Deal.ShortLegSet.EquityLeg.legPrivate.legEffectiveDate" : 1}}], { "allowDiskUse" : true}); 7. On each shard look at the generated profile data. It should look similar to the following. Secondary shard: { "op" : "command" , "ns" : "test.$cmd" , "command" : { "aggregate" : "deal" , "pipeline" : [ ...... ] } .... "millis" : 0 } Primary shard: { "op" : "command" , "ns" : "test.$cmd" , "command" : { "aggregate" : "deal" , "pipeline" : [ { "$mergeCursors" : [ { "host" : "Steves-MacBook-Pro.local:27019" , "id" : NumberLong( "49011188702" ) } ] }, ....... ] }, .... "millis" : 522 }

      An issue has been found where queries which were targeted to a single shard suffered significant performance degradation relative to running the same query against an unsharded collection. The issue was tracked down to an unnecessary mergecursors operation being performed on the primary shard (SERVER-7656). A patched version of 2.6 was subsequently produced to prevent this unnecessary step from being performed however there is an open question as to why the performance of the mergecursrors operation is so bad when it is essentially a noop.

      In order to assist engineering with debugging this issue, I have identified a series of steps than can be followed in order to reliable reproduce this issue on MongoDB 2.6.10 (see 'Steps to reproduce').

      The SERVER-7656 patch will clearly stop this from happening for queries targeted to a single shard however there is an open question as to why merging pre-sorted results from a single shard is taking as long as it is.

        1. data.js
          3 kB
        2. runTest.js
          6 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            steve.dalby@mongodb.com Stephen Dalby
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: