[SERVER-5424] Shell doesn't re-save retrieved integers in an array as integers (converted to 64-bit float) Created: 27/Mar/12  Updated: 10/May/22

Status: Backlog
Project: Core Server
Component/s: JavaScript, Shell
Affects Version/s: 2.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tyler Brock Assignee: DO NOT USE - Backlog - Platform Team
Resolution: Unresolved Votes: 7
Labels: move-sa, platforms-re-triaged
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by RUBY-430 Integers inside array are converted t... Closed
Duplicate
is duplicated by SERVER-41876 db.<collection>.save() cause data typ... Closed
Related
is related to DOCS-6886 Extended JSON docs do not match BSON ... Closed
is related to MONGOSH-1032 NumberInt object wrapper is not displ... Closed
is related to JAVA-832 NumberLong is not properly round trip... Closed
Operating System: ALL
Participants:
Case:

 Description   

# Insert three integers
db.test.insert({ nums: [NumberInt(1), NumberInt(2), NumberInt(3)] })
 
# Check the size
db.test.stats()
{
	"ns" : "mydb.test",
	"count" : 1,
	"size" : 56,
	"avgObjSize" : 56,
	"storageSize" : 4096,
	"numExtents" : 1,
	"nindexes" : 1,
	"lastExtentSize" : 4096,
	"paddingFactor" : 1,
	"flags" : 1,
	"totalIndexSize" : 8176,
	"indexSizes" : {
		"_id_" : 8176
	},
	"ok" : 1
}
 
# Retrieve and save the doc back to the database
> doc = db.test.findOne()
> db.test.save(doc)
 
# Size should be the same if array elements saved as ints.
> db.test.stats()
{
	"ns" : "mydb.test",
	"count" : 1,
	"size" : 116,
	"avgObjSize" : 116,
	"storageSize" : 4096,
	"numExtents" : 1,
	"nindexes" : 1,
	"lastExtentSize" : 4096,
	"paddingFactor" : 1.59,
	"flags" : 1,
	"totalIndexSize" : 8176,
	"indexSizes" : {
		"_id_" : 8176
	},
	"ok" : 1
}

The size increases because they are being re-saved as 64-bit floats and it increases by more than just 6 bytes because of padding factor.



 Comments   
Comment by Steven Vannelli [ 10/May/22 ]

Moving this ticket to the Backlog and removing the "Backlog" fixVersion as per our latest policy for using fixVersions.

Comment by Hassan Faghihi [ 19/Jul/19 ]

i have issue with these data type getting updating.... the complete issue, query and dump data can be found in here:

https://jira.mongodb.org/browse/SERVER-41876

I'm implementing .Net Identity for authentication

Comment by Tyler Brock [ 25/Nov/13 ]

The mongo shell currently converts BSON int32 values to doubles when going from Mongo to V8.

This gives the current implementation two rather nice properties that are depended upon heavily by users of the shell and by the database itself:

  • The doubles are displayed as a series of digits rather than a series of digits wrapped by NumberInt().
  • The doubles can be compared directly.

If we fix the root problem both of these properties disappear.

  • The display of internal numbers that are 32-bit integers is now extremely verbose. (dbstats, rs.status, etc would contain "NumberInt(123)" instead of simply "123".
  • Code depending on comparison of internal numbers / stored data with int32 type will now fail.

The second property loss arises because comparison in Javascript works differently between objects (user defined types) than it does between numbers (an internal type). Comparing two objects of the same type, for example, is done by identity and not by value.

So while it is true that 7 == 7 evaluates as true in JavaScript, NumberInt(7) == NumberInt(7) does not. Many applications depend on the ability to compare these numbers using == and === (including mongod itself) such as when comparing replica set config versions for example.

Further discussion about how to resolve this particular issue is required.

This appears to only be a problem when the integers happen to be in an array but this is simply a case of an optimization obscuring the bug in the most common cases. Most of the time, the mongo shell converts the 32-bit integers to doubles for display but winds up using the original internal BSON representation when saving them back to the database. Therefore, even though in the shell they have the wrong type and are displayed incorrectly it doesn't wind up making a difference.

However, if the integers are in an array, the mongo shell converts them to JavaScript number form internally so that V8 can optimize for processing an array of doubles. V8 is able to process arrays of doubles much faster than arrays of objects when it injects optimized code into the interpreter. In this scenario, the internal BSON representation is invalidated and the objects are re-serialzed as doubles upon saving them to the database.

Comment by Daniel Pasette (Inactive) [ 24/Nov/13 ]

Can't make the proposed change at this time as it changes behavior for many dependent tests and scripts.

Comment by Githook User [ 20/Nov/13 ]

Author:

{u'username': u'ehershey', u'name': u'Ernie Hershey', u'email': u'ernie.hershey@10gen.com'}

Message: Revert "SERVER-5424 use NumberInt function template in mongoToV8Element"

This reverts commit 4a3d5aa6f430530b8a44e581c430176e15da19c2.
Branch: master
https://github.com/mongodb/mongo/commit/89472ab787f3f9ed817a07861080664c61158e97

Comment by Githook User [ 19/Nov/13 ]

Author:

{u'username': u'TylerBrock', u'name': u'Tyler Brock', u'email': u'tyler.brock@gmail.com'}

Message: SERVER-5424 use NumberInt function template in mongoToV8Element

Signed-off-by: Matt Kangas <matt.kangas@mongodb.com>
Branch: master
https://github.com/mongodb/mongo/commit/4a3d5aa6f430530b8a44e581c430176e15da19c2

Comment by Tyler Brock [ 15/Nov/13 ]

This is actually a better example (note: $type: 16 is a 32 bit int and $type: 1 is a double):

> db.dropDatabase( )
{ "dropped" : "test", "ok" : 1 }
> db.test.insert({ nums: [NumberInt(1), NumberInt(2), NumberInt(3)] })
> db.test.find({nums: { $type: 16} })
{ "_id" : ObjectId("52857640a3461b18bcb33e3b"), "nums" : [  1,  2,  3 ] }
> doc = db.test.findOne()
{ "_id" : ObjectId("52857640a3461b18bcb33e3b"), "nums" : [ 1, 2, 3 ] }
> db.test.save(doc)
> db.test.find({nums: { $type: 16} })
> db.test.find({nums: { $type: 1} })
{ "_id" : ObjectId("52857640a3461b18bcb33e3b"), "nums" : [  1,  2,  3 ] }

Comment by Xiaomei Liu [ 14/Feb/13 ]

We had similar issue and the issue is causing retrieving failure after read-update-save operation.

C:\windows\system32>mongo
MongoDB shell version: 2.2.0
connecting to: test

// Create a document with an Int32 field and an Int32 array field:
//

{ a: 1, b: [2] }

> db.testCollection.insert(

{a:NumberInt(1),b:[NumberInt(2)]}

)

// Search for a document with an Int32 'b' field.
// See http://docs.mongodb.org/manual/reference/operator/type/ for a list of $types.
> db.testCollection.findOne({b:{$type:16}})

{ "_id" : ObjectId("511d48c6514abd566bc47327"), "a" : 1, "b" : [ 2 ] }

// Read this document into a local variable.
> var a = db.testCollection.findOne({b:{$type:16}})
> a

{ "_id" : ObjectId("511d48c6514abd566bc47327"), "a" : 1, "b" : [ 2 ] }

// Save the document (without having made any modifications).
> db.testCollection.save(a)

// Search for a document with an Int32 'b' field.
> db.testCollection.findOne({b:{$type:16}})
null

// Search for a document with a Double 'b' field.
> db.testCollection.findOne({b:{$type:1}})

{ "_id" : ObjectId("511d48b0514abd566bc47326"), "a" : 1, "b" : [ 2 ] }

// Search for a document with an Int32 'a' field.
> db.testCollection.findOne({a:{$type:16}})

{ "_id" : ObjectId("511d48c6514abd566bc47327"), "a" : 1, "b" : [ 2 ] }

Conclusion: The Mongo Shell does not properly preserve Int32s stored in arrays.

Generated at Thu Feb 08 03:08:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.