|
Manual testing done. Results summarized below
New stages, accumulators, expressions
As expected, new operators are not available until the mongos is upgraded to 3.2, as the last step in the cycle.
SERVER-14691: $avg of non-numerics now returns null, not 0
As expected, if the $group stage is run on a 3.0 mongod, it will return an average of 0 for non-numeric values. If the $group stage is run on a 3.2 mongod, it will return an average of null.
To be more precise, the following table describes what an $avg of all non-numeric inputs returns in each scenario:
| |
unsharded collection |
sharded collection |
| primary shard for the database is 3.0 |
0 |
0 |
| primary shard for the database is 3.2 |
null |
null |
Note it is only the version of the primary shard for the database that matters for sharded collections, since a $group stage is always run during the merging portion of the aggregation, which (when you have a 3.0 mongos) always happens on the primary shard for that database. If the primary shard for the database is a replica set, your readPreference will determine which node it will run on, which may impact which version of mongod does the merging.
SERVER-6801 $substr now errors if it would split a single UTF-8 character
All worked as expected, an invalid $substr errors when it is run on a 3.2 mongod. Note the $substr expression can be used in the $project stage, which does not force a split of the pipeline, so might be executed in parallel on all the relevant shards, or might be executed during the merging portion, on the primary shard for the database.
SERVER-8141 Arrays are no longer parsed as literal expressions
This is a little more subtle. If you are using a sharded collection, the mongos will parse the pipeline before sending part of it to the relevant shards. If your mongos is on version 3.0, it will parse arrays as literal, then when it serializes the pipeline again to be sent to the shards, it will wrap literal values in a $const expression (functionally the same as $literal). Thus, on sharded collections, you will not observe any of the different parsing behavior for SERVER-8141 until your mongos is upgraded.
For unsharded collections, the mongos does not need to split the pipeline and re-serialize part of it, so it sends the existing command object, as-is, directly to the shard with that collection. In this case, if the mongod that receives it is on version 3.2, it will use the new parsing behavior.
Summary Of Test Execution, Results
The testing involved starting with a sharded cluster where each process was running version 3.0.7, and slowly upgrading it as described in our upgrade docs. The cluster had one mongos, one config server, and two shards. Each shard was a 3-node replica set.
Data setup
_id's have been removed to improve readability.
| Namespace |
Sharded |
Primary shard for database |
Data on shard 1 |
Data on shard 2 |
| shard1.unsharded |
no |
shard1 |
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
None |
| shard1.sharded |
yes |
shard1 |
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
| shard2.unsharded |
no |
shard2 |
None |
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
| shard2.sharded |
yes |
shard2 |
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
{x: 1}
|
{x: 1, y: 2}
|
{x: 2, string: "ǦŔĔĀŢ"}
|
{x: 3, transactions: [{price: "$2"}]}
|
|
Behaviors Tested
| Pipeline run |
Behavior intended to test |
|
|
if a new stage would be recognized. |
[{$project: {
|
x: {$ceil: "$x"}
|
}}]
|
|
if a new expression would be recognized. |
[{$group: {
|
_id: null,
|
x: {$stdDevSamp: "$x"}
|
}}]
|
|
if a new accumulator would be recognized. |
[{$group: {
|
_id: null,
|
x: {$avg: "$non-existent"}
|
}}]
|
|
what an average of no numeric values would return (SERVER-14691) |
[{$project: {
|
x: {$substr: ["$string", 0, 4]}
|
}}]
|
|
if an invalid substring would error, or return an invalid document (SERVER-6801) |
[{$project: {
|
x: {
|
$setIntersection: [
|
"$transactions.price",
|
["$2"]
|
]
|
}
|
}}]
|
|
if "$2" would be treated as a string literal, or as a field path (SERVER-8141) |
Testing Process
- Started a sharded cluster, everything on 3.0.7, verify the following expected behaviors (columns correspond to tests described in 'Behaviors Tested' section)
| $sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
- Upgrade the secondaries of shard1 to 3.2. Verify the following behavior
| ReadPreference |
namespace |
$sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| "primary" |
shard1.unsharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| |
shard1.sharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| |
shard2.sharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| "secondary" |
shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| |
shard2.sharded |
fails |
fails |
fails |
0 |
error |
treated as literal, so matches |
- Upgrade the primary of shard1 to 3.2, observe the following behavior:
| namespace |
$sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| shard2.unsharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| shard2.sharded |
fails |
fails |
fails |
0 |
error |
treated as literal, so matches |
- Upgrade the secondaries of shard2 to 3.2, observe the following behavior:
| ReadPreference |
namespace |
$sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| "primary" |
shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
0 |
invalid UTF-8 character |
treated as literal, so matches |
| |
shard2.sharded |
fails |
fails |
fails |
0 |
error |
treated as literal, so matches |
| "secondary" |
shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard2.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
- Upgrade primary of shard 2 to 3.2, observe the following behavior:
| ReadPreference |
namespace |
$sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| "primary" |
shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard2.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| "secondary" |
shard1.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard1.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
| |
shard2.unsharded |
fails |
fails |
fails |
null |
error |
treated as field path, so no match |
| |
shard2.sharded |
fails |
fails |
fails |
null |
error |
treated as literal, so matches |
- Upgrade the config server, then the mongos to 3.2, observe the following behavior:
| namespace |
$sample |
$ceil |
$stdDevSamp |
$avg |
$substr |
$setIntersection |
| shard1.unsharded |
succeeds |
succeeds |
succeeds |
null |
error |
treated as field path, so no match |
| shard1.sharded |
succeeds |
succeeds |
succeeds |
null |
error |
treated as field path, so no match |
| shard2.unsharded |
succeeds |
succeeds |
succeeds |
null |
error |
treated as field path, so no match |
| shard2.sharded |
succeeds |
succeeds |
succeeds |
null |
error |
treated as field path, so no match |
|