[SERVER-41876] db.<collection>.save() cause data type mismatch Created: 22/Jun/19  Updated: 16/Jul/19  Resolved: 16/Jul/19

Status: Closed
Project: Core Server
Component/s: Admin, JavaScript, Shell
Affects Version/s: 4.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Hassan Faghihi Assignee: Eric Sedor
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File Dumps.rar    
Issue Links:
Duplicate
duplicates SERVER-5424 Shell doesn't re-save retrieved integ... Backlog
Operating System: ALL
Participants:

 Description   

I have data imported using Mongo C# Driver, and at some point i noticed my link are generating certification error since they are not using SSL, so i had to update them somehow, i decide to add both version of links, so i had to update my data in place somehow, and the only way i could find was not update method, but looping through, modification, and save...

In this process i didn't modified any data, or convert them to JSON, i just modified the required field, then when i start the project, i faced internal error for those updated users...

The Int32 field somehow turned to Double, yet the Int64 was intact... i modified object created by DateTimeOffset, it creates an array with two item, an Int64, and an Int32....

But the data type got messed with.

db.humanResource.user.find({
	ProfilePictures: {
		$exists: true
	},
	ProfilePictures: {
		$ne: null
	}
}).forEach(function (e, i) {
	for(var i=0; i< e.ProfilePictures.length; i++){
		 e.ProfilePictures[i].CloudinarySecureUrl = updateHttpToHttps(e.ProfilePictures[i].CloudinaryUrl);
	}
	db.humanResource.user.save(e);
})



 Comments   
Comment by Eric Sedor [ 05/Jul/19 ]

Hi deadmann; thanks for the additional information. The behavior you are seeing is the result of SERVER-5424, which specifically impacts integers in arrays, such as LockoutEndDateUtc in the provided data.

I'd like to suggest watching SERVER-5424 for updates and adding your input to the discussion there.

Comment by Hassan Faghihi [ 25/Jun/19 ]

Luckily i have test data, as my data are not production yet...

 

Here's the rar file

 

Dumps.rar

Comment by Eric Sedor [ 24/Jun/19 ]

With that in mind, are you able to provide a bsondump of both an affected document and an un-affected document?

You can upload them to this secure upload portal. Files uploaded here are visible only to MongoDB employees and are deleted after some time.

It will also help if you can list the field names that were impacted.

Comment by Hassan Faghihi [ 24/Jun/19 ]
  • data was imported via the C# driver -> yes
  • data migration to change ProfilePictures.CloudinarySecureUrl was run in the shell -> yes
  • understand that data types changed for fields you didn't intend to modify? -> yes, Note i notice normal int field do not own any kind of schematic annotation, i'm not sure where i see that, but i'm sure some of numerical data where marked as long and double..., when i changed them inside 3T studio (which became double, and i returned it to int32) everything start working. also Int64 was intact (which as i said had schematic annotations)
  • Since clients in different languages can and do deserialize/serialize data types differently, this is likely the cause of the behavior you see. -> Well the shell is provided by mango developer, and it's unlikely to be someone else writing or language mismatch.. also although json de/serializer might stop writing the data types, but BSON de/serializer knows everything, and other should be considered plain int32 objects... also 0 is unlickly to be converted to 0.0

well i need a fast mass update, and it was unlickly that update be capable to override a field data using it's own data or it's sibling data. so i found this code, which inside foreEach while the cursor is running update whole document... i'm wonder if update is runnable too.

Comment by Eric Sedor [ 24/Jun/19 ]

Hi deadmann,

Do I understand correctly that the data was imported via the C# driver and the data migration to change ProfilePictures.CloudinarySecureUrl was run in the shell? Do I also understand that data types changed for fields you didn't intend to modify?

If so, this is somewhat expected because of the use of .save(), which performs full document updates by _id. The code you provided pulls the whole document to the client, deserializes it, then changes the field, then re-serializes it and sends it back to MongoDB in its entirety to overwrite the original. Since clients in different languages can and do deserialize/serialize data types differently, this is likely the cause of the behavior you see.

For modifications to specific fields, you can limit the risk of data migration scripts by using update operators like $set to perform updates on the database side rather than the client side.

Does this make sense? Let me know if I've misunderstood.

Generated at Thu Feb 08 04:58:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.