[SERVER-6166] consider expanding to wider data type on math overflow Created: 21/Jun/12  Updated: 28/Oct/15  Resolved: 27/Jul/12

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 2.2.0-rc1

Type: Question Priority: Major - P3
Reporter: Aaron Staple Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-6510 $sum can cause int overflow Closed
Related
related to SERVER-6203 Aggregation operators should have wel... Backlog
related to SERVER-6197 $avg is calculated using integer math... Closed
is related to SERVER-5239 $project should have numeric type coe... Closed
Participants:

 Description   

Observed behavior: The result of multiplying two 32 bit integers is a 32 bit integer, even if there is an overflow.
Proposed behavior: Rather than overflow, an alternative numeric type will be used to store the result when possible.

c = db.c;
c.drop();
 
c.save( {} );
assert.eq( NumberLong( '4611686018427390000' ),
           // Current behavior on overflow is for the multiplication result to be zero.
           c.aggregate( { $project:{ a:{ $multiply:[ NumberInt( '2147483648' ), NumberInt( '2147483648' )\
 ] } } } ).result[ 0 ].a );

Consider addition and subtraction overflow as well. For numeric addition and subtraction, adding two 32 bit ints will produce a 32 bit int. Currently division is implemented to always return a double.

Also consider for accumulators like $sum.



 Comments   
Comment by auto [ 27/Jul/12 ]

Author:

{u'date': u'2012-07-26T07:29:12-07:00', u'email': u'dannenberg.matt@gmail.com', u'name': u'Matt Dannenberg'}

Message: SERVER-6166 cast from int to long on overflow agg

this was started in the past couple commits, but this covers the final couple cases
Branch: master
https://github.com/mongodb/mongo/commit/b5a215f3602ff054025732d082453bd503757792

Comment by auto [ 27/Jul/12 ]

Author:

{u'date': u'2012-07-23T14:27:55-07:00', u'email': u'dannenberg.matt@gmail.com', u'name': u'Matt Dannenberg'}

Message: SERVER-6275 SERVER-6197 use double for $avg

also do not count non-numeric types in $avg
also part of SERVER-6166 up convert from int to long on $sum
Branch: master
https://github.com/mongodb/mongo/commit/186c5a411c3a1e616f7e876820a13af46d6faa4c

Comment by auto [ 27/Jul/12 ]

Author:

{u'date': u'2012-07-24T06:59:40-07:00', u'email': u'dannenberg.matt@gmail.com', u'name': u'Matt Dannenberg'}

Message: SERVER-6570 disallow strings in $add in aggro

also started SERVER-6166 (ints overflowing to long in math) by doing the add portion
redo testall.js p20 when SERVER-6195 is done
Branch: master
https://github.com/mongodb/mongo/commit/d3b12c40a91d20dc5b86e40402308c7b2320f9a5

Comment by Mathias Stearn [ 20/Jul/12 ]

I'm going to use roughly the same rules as $inc updates. If anything is a double output is double. If mix of int and long, output is long. If only int, output is int unless it would overflow, else promote to long.

In addition to $sum, this also effects $avg, which absolutely should never overflow since it can lead to misleadingly low results and is always output as a double anyway.

Comment by Andy Schwerin [ 20/Jul/12 ]

Would it be better to let users coerce input types to govern output types?

Comment by Andy Schwerin [ 20/Jul/12 ]

What is your spec for this? If a user wishes to observe overflow rather than type widening, how will the user express that?

Comment by Mathias Stearn [ 19/Jul/12 ]

moving to 2.2-rc1 since this is really easy to fix

Comment by Andy Schwerin [ 23/Jun/12 ]

The thing about punting this to 2.3.0 is that I'm concerned that someone will come to depend on the overflow behavior, and we'll be stuck with it. If we get around to allowing aggregation pipelines to output into collections, we'll probably have to support operators to coerce types.

Comment by Aaron Staple [ 21/Jun/12 ]

That sounds good to me. I tried to make the initial description pretty general and don't have strong opinions on the specifics.

Comment by Andy Schwerin [ 21/Jun/12 ]

I'd be amenable to widening to 64-bit integer, but generally opposed to widening to precision-losing double.

Generated at Thu Feb 08 03:10:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.