[SERVER-21201] Increase logging severity for Assertion: 28556:Didn't find RecordId in WiredTigerRecordStore Created: 29/Oct/15 Updated: 06/Dec/22 Resolved: 12/Apr/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging |
| Affects Version/s: | 3.0.5 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Mike Zraly | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Duplicate | Votes: | 2 |
| Labels: | WTplaybook | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
We are experiencing an error where we can find some records with a query on the secondary but not on the primary. We are still working on a reproducible test case for this and will file a bug once we have once. However. When a record can't be found the only indication we have in mongodb.log is an assertion log with severity I(nformational):
There were no messages with Warning or Error severity to report the failure to find the record. This failure can lead to loss of data, and warrants higher severity, and probably a dedicated error message. |
| Comments |
| Comment by Eric Milkie [ 12/Apr/19 ] | ||||||||||||||||||||||
|
The message became a fatal error in | ||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 03/Feb/16 ] | ||||||||||||||||||||||
|
Thanks for the update mzraly. In your case the assertion is triggered by a distinct() command; do you still use distinct() in your application? If yes then I suspect an interaction with an index build, but this is just a wild guess. Anyway, I'm going to send this ticket to the right team to consider the improvement to the log level. Feel free to post here again if you see the assertion reappear. nifan, I'm going to open a separate ticket on your behalf – if you're still seeing this issue there are some options we can explore. Regards, | ||||||||||||||||||||||
| Comment by Mike Zraly [ 03/Feb/16 ] | ||||||||||||||||||||||
|
Sorry for the much-delayed reply. We have not reproduced this issue since we upgraded to 3.0.7. I'm afraid I don't have access to the old logs anymore, but I believe with 60% certainty that the secondary where we found the documents we did not see on the primary has once been the primary. We do delete documents though so it's possible that we didn't lose a copy of the record but rather a delete. I'm sorry I don't have any more details to provide. | ||||||||||||||||||||||
| Comment by nifan [ 15/Dec/15 ] | ||||||||||||||||||||||
|
Sorry we are running:
| ||||||||||||||||||||||
| Comment by Kelsey Schubert [ 15/Dec/15 ] | ||||||||||||||||||||||
|
Hi nifan, It would help if you could provide us with a few more details.
Thank you, | ||||||||||||||||||||||
| Comment by nifan [ 14/Dec/15 ] | ||||||||||||||||||||||
|
We seem to have the same problem.
| ||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 04/Dec/15 ] | ||||||||||||||||||||||
|
mzraly, we're discussing internally how you may have arrived to this situation. Have you been able to reproduce this again? Is there any other information about your deployment that you can provide that can help the investigation? For example, the redacted logs do not allow us to see which node became primary when, so when you mention you found documents on a secondary that were not there on a primary, was that secondary node a primary before? Thanks, | ||||||||||||||||||||||
| Comment by Mike Zraly [ 06/Nov/15 ] | ||||||||||||||||||||||
|
Redacted log file containing several examples of Assertion 28556. It looks like there were some early cases where a query was made to the instance before background index creation for the appropriate query index completed. However there are many instances of this assertion after the index creation completed for all collections. | ||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 29/Oct/15 ] | ||||||||||||||||||||||
|
Thanks for your report mzraly. Could you please upload full logs for this node that include the messages you included in the description? This will help us put this particular assertion in context. Thanks, |