[SERVER-3717] loss of precision for long long values in v1 index Created: 29/Aug/11  Updated: 11/Jul/16  Resolved: 01/Sep/11

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: 2.0.0-rc1

Type: Bug Priority: Major - P3
Reporter: Aaron Staple Assignee: Dwight Merriman
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-3719 woCompare does not provide a total or... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

I think this
double d = (double) n;
if( d != n ) {

needs to be this

double d = (double) n;
if( (long long)d != n ) {

Test forthcoming.



 Comments   
Comment by auto [ 31/Aug/11 ]

Author:

{u'login': u'dwight', u'name': u'Dwight', u'email': u'dwight@10gen.com'}

Message: SERVER-3717 last commit wasn't quite right
Branch: master
https://github.com/mongodb/mongo/commit/cba5d568e9e48398a0fa34aefb4904be464e6570

Comment by auto [ 31/Aug/11 ]

Author:

{u'login': u'dwight', u'name': u'Dwight', u'email': u'dwight@10gen.com'}

Message: SERVER-3717
fix check if value to large to represent as a long in KeyV1
for some reason was failing on win32. not sure why. visual studio 2008 issue?
regardless, the old implementation was a bad idea when going from machine/build
to machine as then the results could vary if one was compiled with strict
floating point and the other non-strict.
Branch: master
https://github.com/mongodb/mongo/commit/1c5b750fdc624cbbade7e33ccc89c3e55faf5fc5

Comment by auto [ 30/Aug/11 ]

Author:

{u'login': u'dwight', u'name': u'dwight', u'email': u'dwight@10gen.com'}

Message: SERVER-3717 enable test
Branch: master
https://github.com/mongodb/mongo/commit/fff7a4972ea821104707f0e5008e615030fea6b6

Comment by Dwight Merriman [ 30/Aug/11 ]

despite the comparision issue still being open (3719), i believe this fixes the covered index loss of precision case this ticket covers.

Comment by Dwight Merriman [ 30/Aug/11 ]

there was a bug, fixed here: https://github.com/mongodb/mongo/commit/9fd22cb86de06cbb4b710e4d399f81b9a709f4f3

however as far as i can tell this has never worked including for unindexed cases (base woCompare method). so it still doesn't work but the problem is more general and not new. aaron if that isn't correct let me know. fixing the whole thing will be too large a change and should be in 2.1. i have added a test case to dbtests/jsobj.cpp which i will push post 2.0. and when that is done it will have to be in a new index version #.

Comment by auto [ 29/Aug/11 ]

Author:

{u'login': u'astaple', u'name': u'Aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-3717 test
Branch: master
https://github.com/mongodb/mongo/commit/a406f1b8f4adba88b876417df5a8b081695c05a9

Generated at Thu Feb 08 03:03:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.