[SERVER-22316] MongoDB.Driver.WriteConcernException in specific update queries Created: 26/Jan/16  Updated: 20/Jun/16  Resolved: 20/Jun/16

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rodrigo Cetera Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: Ubuntu 15.04
Proc: Intel Core i7-4790 Quad-Core CPU @ 3.60GHz
RAM: 32GB
Profile: 1 slow:100


Attachments: Text File error.txt     Text File exception.txt     Text File query.txt     Text File storage.txt    
Operating System: Linux
Participants:

 Description   

When trying to make a particular update query (query.txt) over a stored document (storage.txt) the following error appear:(error.txt)

WriteConcern detected an error 'assertion src/mongo/db/storage/mmap_v1/btree/key.cpp:608'. (Response was { "connectionId" : 1006, "err" : "assertion src/mongo/db/storage/mmap_v1/btree/key.cpp:608", "n" : 0, "ok" : 1.0 }). 
Type: MongoDB.Driver.WriteConcernException

I realized that this only happens when the query have the exact field name "size_of_raw_data" with value 512. If i change the value to 513 or 511 or whatever else, the query executes in normal way.



 Comments   
Comment by Kelsey Schubert [ 20/Jun/16 ]

Hi kdeltared,

Thanks for getting back to us with the answers to my questions. From your responses, I do not see a clear explanation for the corruption you have observed. It may have been the result of an unclean shutdown some time in the past or faulty disk drives. Unfortunately, as I mentioned previously determining the root cause of this corruption is very challenging without a reproduction. If this issue persists after you have ensured the integrity of your disk drives, please let us know and we will continue to investigate this issue.

Thank you for your help,
Thomas

Comment by Ramon Fernandez Marina [ 18/May/16 ]

kdeltared, looks like your reply fell through the cracks – my apologies for that. I've reopened the ticket to take a closer look.

Comment by Rodrigo Cetera [ 08/Mar/16 ]

Hi, sorry for the delay.
1. There is no replica set.
2. Nothing else logged.
3. Journaling is activated
4. I´m using 5 local HDDs in Software Raid 5 configuration.
5. Files never moved or coppied.
6. Never been restored.
7. By the time, there is no BackUp

Right now, we are thinking to migrate to MongoDB 3.2 with WiredTiger and make other changes like SDDs disk and raid configuration, so we are going to rebuild the DB from scratch. Anyway I will execute repairDatabase in maintenance cycle to see what happens.
For the moment, changing the field name help to keep things in production, but would be interesting to know what really happened.
I will post the results from repairDatabese when i can run it.

Thank You,
Rodrigo

Comment by Ramon Fernandez Marina [ 05/Mar/16 ]

kdeltared, we haven't heard back from you for some time, so I'm going to close this ticket. If this is still an issue for you please provide the information requested above by Thomas so we can investigate further.

Regards,
Ramón.

Comment by Kelsey Schubert [ 02/Feb/16 ]

Hi kdeltared,

This assertion failure generally indicates that some or all of the data files have become corrupt in some way. It's not clear if the corruption is in the index or the data itself, and in cases like this, it's very difficult to be confident that the corruption is isolated beyond the file level.

To help us understand what's going on here, I've assembled a list of routine questions about data storage and the configuration of your environment. We can use this to begin debugging the underlying issue, but beware in these sorts of situations it can be difficult to understand the cause of the corruption without a straightforward reproduction.

  1. If this node is part of a replicaset, are other members affected?
  2. Around this operation, were there any other server errors logged?
  3. Are you using journaling?
  4. What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?
  5. Have you manipulated (coppied or moved) the underling database files?
  6. Have you ever restored this instance from backups?
  7. What method do you use to create backups?

I would recommend a clean resync from a node that is not affected. If that is not possible, I would recommend executing repairDatabase.

Additionally, please consider doing a thorough integrity check of the affected node's disk drives. If the assertion failures continue to persist, you may need to replace them.

Thank you,
Thomas

Comment by Rodrigo Cetera [ 28/Jan/16 ]

Ramon, Im am using MongoDB 3.0.9 with MMAP engine.
I attached the profile log for the query if it is of help. I dont have any script.
Indexes are follows:
id : 1
file_id : hashed
hash.md5 : hashed
hash.sha2 : hashed
mime_type : hashed
particular_header.file_entropy : 1
particular_header.headers.file_header.TimeDateStamp : 1
particular_header.imports.functions : 1
particular_header.imports.lib : 1
particular_header.sections.md5 : 1
particular_header.sections.name : 1
particular_header.sections.sha1 : 1
particular_header.sections.sha2 : 1
particular_header.sections.size_of_raw_data : 1
size: 1

Thank you

Comment by Ramon Fernandez Marina [ 26/Jan/16 ]

kdeltared, what version of MongoDB are you using? What indexes are defined on this collection?

I imported the document from storage.txt on a MongoDB 3.2.1 instance and I was able to successfully run the query. Can you please provide more information that can help us investigate? If you have a repro script that would be even better.

Thanks,
Ramón.

Generated at Thu Feb 08 04:00:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.