Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32839

Aggregation sorts on multiple array fields produce error in 3.6

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.0, 3.7.1
    • Component/s: Aggregation Framework
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Hide

      Sample dataset:

      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41417") }, "creationDate" : ISODate("2018-01-22T18:37:13.643Z"), "location" : { "type" : "Point", "coordinates" : [ -73.41825120678124, 41.11954344761012 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 2, "b" : 0, "s" : "XYZ_2", "d" : ISODate("2065-11-12T14:57:25.761Z"), "e" : 2, "mixed" : 2, "l" : [ 2, 3 ], "m" : { "i" : 2 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.659Z") }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41418") }, "creationDate" : ISODate("2018-01-22T18:37:13.644Z"), "location" : { "type" : "Point", "coordinates" : [ -73.37204671642996, 41.685165896010886 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 3, "b" : 1, "s" : "XYZ_3", "d" : ISODate("2036-05-22T06:29:17.274Z"), "f" : 3, "mixed" : "a", "l" : [ 3, 4 ], "m" : { "i" : 3 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.660Z") }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41419") }, "creationDate" : ISODate("2018-01-22T18:37:13.645Z"), "location" : { "type" : "Point", "coordinates" : [ -73.53230022652237, 41.99925690167811 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 4, "b" : 0, "s" : "XYZ_4", "d" : ISODate("2040-09-27T12:39:26.191Z"), "e" : 4, "mixed" : 4, "l" : [ 4, 5 ], "m" : { "i" : 4 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.661Z") }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141a") }, "creationDate" : ISODate("2018-01-22T18:37:13.647Z"), "location" : { "type" : "Point", "coordinates" : [ -73.37406135644643, 41.20800859893658 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 5, "b" : 1, "s" : "XYZ_5", "d" : ISODate("2032-01-12T02:54:20.554Z"), "mixed" : 5, "l" : [ 5, 6 ], "m" : { "i" : 5 } } }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141b") }, "creationDate" : ISODate("2018-01-22T18:37:13.648Z"), "location" : { "type" : "Point", "coordinates" : [ -73.23532883469254, 41.82323919775319 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 6, "b" : 0, "s" : "XYZ_6", "d" : ISODate("2040-06-19T23:57:21.058Z"), "e" : 6, "f" : 6, "mixed" : "a", "l" : [ 6, 7 ], "m" : { "i" : 6 } } }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141c") }, "creationDate" : ISODate("2018-01-22T18:37:13.649Z"), "location" : { "type" : "Point", "coordinates" : [ -73.836244603408, 41.10642459488787 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 7, "b" : 1, "s" : "XYZ_7", "d" : ISODate("2023-06-04T05:10:46.560Z"), "mixed" : 7, "l" : [ 7, 8 ], "m" : { "i" : 7 } } }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141d") }, "creationDate" : ISODate("2018-01-22T18:37:13.650Z"), "location" : { "type" : "Point", "coordinates" : [ -73.25802811955722, 41.6346098603488 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 8, "b" : 0, "s" : "XYZ_8", "d" : ISODate("2045-12-15T12:08:01.250Z"), "e" : 8, "mixed" : 8, "l" : [ 8, 9 ], "m" : { "i" : 8 } } }
      
      { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141e") }, "creationDate" : ISODate("2018-01-22T18:37:13.651Z"), "location" : { "type" : "Point", "coordinates" : [ -73.17641662154917, 41.15470847481475 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 9, "b" : 1, "s" : "XYZ_9", "d" : ISODate("2060-01-04T02:18:06.486Z"), "f" : 9, "mixed" : "a", "l" : [ 9, 10 ], "m" : { "i" : 9 } } }
      

      Indexes:

      > db.drops.getIndexes()
      [
      	{
      		"v" : 2,
      		"key" : {
      			"_id" : 1
      		},
      		"name" : "_id_",
      		"ns" : "drops.drops"
      	},
      	{
      		"v" : 2,
      		"key" : {
      			"location" : "2dsphere"
      		},
      		"name" : "location_2dsphere",
      		"ns" : "drops.drops",
      		"background" : true,
      		"2dsphereIndexVersion" : 3
      	},
      	{
      		"v" : 2,
      		"key" : {
      			"_id.b" : 1,
      			"_id.d" : 1
      		},
      		"name" : "_id.b_1__id.d_1",
      		"ns" : "drops.drops",
      		"background" : true
      	},
      	{
      		"v" : 2,
      		"key" : {
      			"_id.d" : 1
      		},
      		"name" : "_id.d_1",
      		"ns" : "drops.drops",
      		"background" : true
      	},
      	{
      		"v" : 2,
      		"key" : {
      			"_id.b" : 1,
      			"creationDate" : -1
      		},
      		"name" : "_id.b_1_creationDate_-1",
      		"ns" : "drops.drops",
      		"background" : true
      	},
      	{
      		"v" : 2,
      		"key" : {
      			"_id.b" : 1,
      			"fhash" : 1
      		},
      		"name" : "_id.b_1_fhash_1",
      		"ns" : "drops.drops",
      		"background" : true,
      		"sparse" : true
      	}
      ]
      

      Aggregation

      i=[{ "$match" : { "aZUQ.a" : { "$gt" : 1} , "_id.b" : ObjectId("5a662f597f3b985a7cd41414")}},
      { "$project" : { "a" : "$aZUQ.a" , "month_creationDate" : { "$month" : "$creationDate"}}},
      { "$group" : { "_id" : { "a" : "$a"} , "a" : { "$addToSet" : "$a"} , "month" : { "$addToSet" : "$month_creationDate"}}},
      { "$sort" : { "month" : 1 , "a" : 1}},
      { "$limit" : 100}]
      

      db.drops.aggregate([i[0], i[1], i[2]])

      { "_id" : { "a" : 9 }, "a" : [ 9 ], "month" : [ 1 ] },
      { "_id" : { "a" : 8 }, "a" : [ 8 ], "month" : [ 1 ] },
      { "_id" : { "a" : 7 }, "a" : [ 7 ], "month" : [ 1 ] },
      { "_id" : { "a" : 6 }, "a" : [ 6 ], "month" : [ 1 ] },
      { "_id" : { "a" : 4 }, "a" : [ 4 ], "month" : [ 1 ] },
      { "_id" : { "a" : 3 }, "a" : [ 3 ], "month" : [ 1 ] },
      { "_id" : { "a" : 5 }, "a" : [ 5 ], "month" : [ 1 ] },
      { "_id" : { "a" : 2 }, "a" : [ 2 ], "month" : [ 1 ] }
      

      Adding the sort line will fail:

      db.drops.aggregate([i[0], i[1], i[2],i[3]])`
      >
      2018-01-22T13:38:07.220-0500 E QUERY    [thread1] Error: command failed: {
      	"ok" : 0,
      	"errmsg" : "cannot sort with keys that are parallel arrays",
      	"code" : 2,
      	"codeName" : "BadValue"
      } : aggregate failed :
      _getErrorWithCode@src/mongo/shell/utils.js:25:13
      doassert@src/mongo/shell/assert.js:16:14
      assert.commandWorked@src/mongo/shell/assert.js:403:5
      DB.prototype._runAggregate@src/mongo/shell/db.js:260:9
      DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1212:12
      @(shell):1:1
      
      Show
      Sample dataset: { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41417") }, "creationDate" : ISODate("2018-01-22T18:37:13.643Z"), "location" : { "type" : "Point", "coordinates" : [ -73.41825120678124, 41.11954344761012 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 2, "b" : 0, "s" : "XYZ_2", "d" : ISODate("2065-11-12T14:57:25.761Z"), "e" : 2, "mixed" : 2, "l" : [ 2, 3 ], "m" : { "i" : 2 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.659Z") } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41418") }, "creationDate" : ISODate("2018-01-22T18:37:13.644Z"), "location" : { "type" : "Point", "coordinates" : [ -73.37204671642996, 41.685165896010886 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 3, "b" : 1, "s" : "XYZ_3", "d" : ISODate("2036-05-22T06:29:17.274Z"), "f" : 3, "mixed" : "a", "l" : [ 3, 4 ], "m" : { "i" : 3 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.660Z") } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd41419") }, "creationDate" : ISODate("2018-01-22T18:37:13.645Z"), "location" : { "type" : "Point", "coordinates" : [ -73.53230022652237, 41.99925690167811 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 4, "b" : 0, "s" : "XYZ_4", "d" : ISODate("2040-09-27T12:39:26.191Z"), "e" : 4, "mixed" : 4, "l" : [ 4, 5 ], "m" : { "i" : 4 } }, "lastEditDate" : ISODate("2018-01-22T18:37:14.661Z") } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141a") }, "creationDate" : ISODate("2018-01-22T18:37:13.647Z"), "location" : { "type" : "Point", "coordinates" : [ -73.37406135644643, 41.20800859893658 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 5, "b" : 1, "s" : "XYZ_5", "d" : ISODate("2032-01-12T02:54:20.554Z"), "mixed" : 5, "l" : [ 5, 6 ], "m" : { "i" : 5 } } } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141b") }, "creationDate" : ISODate("2018-01-22T18:37:13.648Z"), "location" : { "type" : "Point", "coordinates" : [ -73.23532883469254, 41.82323919775319 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 6, "b" : 0, "s" : "XYZ_6", "d" : ISODate("2040-06-19T23:57:21.058Z"), "e" : 6, "f" : 6, "mixed" : "a", "l" : [ 6, 7 ], "m" : { "i" : 6 } } } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141c") }, "creationDate" : ISODate("2018-01-22T18:37:13.649Z"), "location" : { "type" : "Point", "coordinates" : [ -73.836244603408, 41.10642459488787 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 7, "b" : 1, "s" : "XYZ_7", "d" : ISODate("2023-06-04T05:10:46.560Z"), "mixed" : 7, "l" : [ 7, 8 ], "m" : { "i" : 7 } } } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141d") }, "creationDate" : ISODate("2018-01-22T18:37:13.650Z"), "location" : { "type" : "Point", "coordinates" : [ -73.25802811955722, 41.6346098603488 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 8, "b" : 0, "s" : "XYZ_8", "d" : ISODate("2045-12-15T12:08:01.250Z"), "e" : 8, "mixed" : 8, "l" : [ 8, 9 ], "m" : { "i" : 8 } } } { "_id" : { "b" : ObjectId("5a662f597f3b985a7cd41414"), "d" : ObjectId("5a662f597f3b985a7cd4141e") }, "creationDate" : ISODate("2018-01-22T18:37:13.651Z"), "location" : { "type" : "Point", "coordinates" : [ -73.17641662154917, 41.15470847481475 ] }, "creatorId" : "i000000000000000000000001", "aZUQ" : { "a" : 9, "b" : 1, "s" : "XYZ_9", "d" : ISODate("2060-01-04T02:18:06.486Z"), "f" : 9, "mixed" : "a", "l" : [ 9, 10 ], "m" : { "i" : 9 } } } Indexes: > db.drops.getIndexes() [ { "v" : 2, "key" : { "_id" : 1 }, "name" : "_id_", "ns" : "drops.drops" }, { "v" : 2, "key" : { "location" : "2dsphere" }, "name" : "location_2dsphere", "ns" : "drops.drops", "background" : true, "2dsphereIndexVersion" : 3 }, { "v" : 2, "key" : { "_id.b" : 1, "_id.d" : 1 }, "name" : "_id.b_1__id.d_1", "ns" : "drops.drops", "background" : true }, { "v" : 2, "key" : { "_id.d" : 1 }, "name" : "_id.d_1", "ns" : "drops.drops", "background" : true }, { "v" : 2, "key" : { "_id.b" : 1, "creationDate" : -1 }, "name" : "_id.b_1_creationDate_-1", "ns" : "drops.drops", "background" : true }, { "v" : 2, "key" : { "_id.b" : 1, "fhash" : 1 }, "name" : "_id.b_1_fhash_1", "ns" : "drops.drops", "background" : true, "sparse" : true } ] Aggregation i=[{ "$match" : { "aZUQ.a" : { "$gt" : 1} , "_id.b" : ObjectId("5a662f597f3b985a7cd41414")}}, { "$project" : { "a" : "$aZUQ.a" , "month_creationDate" : { "$month" : "$creationDate"}}}, { "$group" : { "_id" : { "a" : "$a"} , "a" : { "$addToSet" : "$a"} , "month" : { "$addToSet" : "$month_creationDate"}}}, { "$sort" : { "month" : 1 , "a" : 1}}, { "$limit" : 100}] db.drops.aggregate([i[0], i[1], i[2]]) { "_id" : { "a" : 9 }, "a" : [ 9 ], "month" : [ 1 ] }, { "_id" : { "a" : 8 }, "a" : [ 8 ], "month" : [ 1 ] }, { "_id" : { "a" : 7 }, "a" : [ 7 ], "month" : [ 1 ] }, { "_id" : { "a" : 6 }, "a" : [ 6 ], "month" : [ 1 ] }, { "_id" : { "a" : 4 }, "a" : [ 4 ], "month" : [ 1 ] }, { "_id" : { "a" : 3 }, "a" : [ 3 ], "month" : [ 1 ] }, { "_id" : { "a" : 5 }, "a" : [ 5 ], "month" : [ 1 ] }, { "_id" : { "a" : 2 }, "a" : [ 2 ], "month" : [ 1 ] } Adding the sort line will fail: db.drops.aggregate([i[0], i[1], i[2],i[3]])` > 2018-01-22T13:38:07.220-0500 E QUERY [thread1] Error: command failed: { "ok" : 0, "errmsg" : "cannot sort with keys that are parallel arrays", "code" : 2, "codeName" : "BadValue" } : aggregate failed : _getErrorWithCode@src/mongo/shell/utils.js:25:13 doassert@src/mongo/shell/assert.js:16:14 assert.commandWorked@src/mongo/shell/assert.js:403:5 DB.prototype._runAggregate@src/mongo/shell/db.js:260:9 DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1212:12 @(shell):1:1

      Original Summary

      Aggregation sorting error in 3.5, 3.6. No error in 3.4

      Original Description

      We have some generated aggregation queries. I've verified that our tests fail with 3.5, 3.6 and 3.7 server (testing on OSX, single instance environment) but succeed with 3.4 (and I'm assuming earlier versions since we were on 3.2 and this has never caused us a problem).The queries are being generated from the java driver but fail identically in the shell.

      This happens in an aggregation sort step. Given an aggregation whose pipeline steps return the following:

       { "_id" : { "a" : 2 }, "a" : [ 2 ], "month" : [ 1 ] }
       { "_id" : { "a" : 3 }, "a" : [ 3 ], "month" : [ 1 ] }
       { "_id" : { "a" : 4 }, "a" : [ 4 ], "month" : [ 1 ] }
       { "_id" : { "a" : 5 }, "a" : [ 5 ], "month" : [ 1 ] }
       { "_id" : { "a" : 6 }, "a" : [ 6 ], "month" : [ 1 ] }
       { "_id" : { "a" : 7 }, "a" : [ 7 ], "month" : [ 1 ] }
       { "_id" : { "a" : 8 }, "a" : [ 8 ], "month" : [ 1 ] }
       { "_id" : { "a" : 9 }, "a" : [ 9 ], "month" : [ 1 ] }
      

      adding a step of the form {"$sort":{"month":1, "a":1}}

      Throws the error:

      2018-01-22T13:38:07.220-0500 E QUERY    [thread1] Error: command failed: {
      	"ok" : 0,
      	"errmsg" : "cannot sort with keys that are parallel arrays",
      	"code" : 2,
      	"codeName" : "BadValue"
      } : aggregate failed :
      _getErrorWithCode@src/mongo/shell/utils.js:25:13
      doassert@src/mongo/shell/assert.js:16:14
      assert.commandWorked@src/mongo/shell/assert.js:403:5
      DB.prototype._runAggregate@src/mongo/shell/db.js:260:9
      DBCollection.prototype.aggregate@src/mongo/shell/collection.js:1212:12
      @(shell):1:1
      

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            sbenjamin Steven Benjamin
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: