[SERVER-8033] Sharded agg should only send needed fields from shards to mongos Created: 26/Dec/12  Updated: 11/Jul/16  Resolved: 21/Jan/14

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: 2.5.5

Type: Improvement Priority: Major - P3
Reporter: Mathias Stearn Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Backwards Compatibility: Fully Compatible
Participants:

 Description   

This currently happens automatically if the split-point is a $group, but not if it is a $sort, $skip, or $limit. If any of the mongos-only stages return EXHAUSTIVE from getDependencies(), but none of the shard stages do, then a $project should be added to the end of the shard stages.

Advantages:

  • reduce mongod -> mongos network traffic
  • allow mongod to reify only needed fields
  • allow more documents to be output from a shard before hitting 16MB limit.


 Comments   
Comment by Githook User [ 21/Jan/14 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-8033 Sharded agg should only send needed fields from shards to mongos

*** BEFORE ***
 
> db.foo.aggregate([{$sort: {score:1}}, {$limit: 10}, {$project: {_id: 1}}], {explain:true})
{
    "splitPipeline" : {
        "shardsPart" : [
            {
                "$sort" : {
                    "sortKey" : {
                        "score" : 1
                    },
                    "limit" : NumberLong(10)
                }
            }
        ],
        "mergerPart" : [
            {
                "$sort" : {
                    "sortKey" : {
                        "score" : 1
                    },
                    "mergePresorted" : true,
                    "limit" : NumberLong(10)
                }
            },
            {
                "$project" : {
                    "_id" : true
                }
            }
        ]
    },
    "shards" : {
        "shard0000" : {
            "host" : "localhost:30001",
            "stages" : [
                {
                    "$cursor" : {
                        "query" : {
 
                        },
                        "indexOnly" : false,
                        "cursorType" : "BasicCursor"
                    }
                },
                {
                    "$sort" : {
                        "sortKey" : {
                            "score" : 1
                        },
                        "limit" : NumberLong(10)
                    }
                }
            ]
        }
    },
    "ok" : 1
}
 
*** AFTER ***
 
> db.foo.aggregate([{$sort: {score:1}}, {$limit: 10}, {$project: {_id: 1}}], {explain:true})
{
    "splitPipeline" : {
        "shardsPart" : [
            {
                "$sort" : {
                    "sortKey" : {
                        "score" : 1
                    },
                    "limit" : NumberLong(10)
                }
            },
            {
                "$project" : {
                    "score" : true,
                    "_id" : true
                }
            }
        ],
        "mergerPart" : [
            {
                "$sort" : {
                    "sortKey" : {
                        "score" : 1
                    },
                    "mergePresorted" : true,
                    "limit" : NumberLong(10)
                }
            },
            {
                "$project" : {
                    "_id" : true
                }
            }
        ]
    },
    "shards" : {
        "shard0000" : {
            "host" : "localhost:30001",
            "stages" : [
                {
                    "$cursor" : {
                        "query" : {
 
                        },
                        "fields" : {
                            "score" : 1,
                            "_id" : 1
                        },
                        "indexOnly" : false,
                        "cursorType" : "BasicCursor"
                    }
                },
                {
                    "$sort" : {
                        "sortKey" : {
                            "score" : 1
                        },
                        "limit" : NumberLong(10)
                    }
                },
                {
                    "$project" : {
                        "score" : true,
                        "_id" : true
                    }
                }
            ]
        }
    },
    "ok" : 1
}

Branch: master
https://github.com/mongodb/mongo/commit/c9fc8a468e1fa9d6421ef35f5a23db0e0f014b4f

Generated at Thu Feb 08 03:16:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.