[SERVER-63881] Input/output error Created: 22/Feb/22  Updated: 10/Jun/22  Resolved: 17/Mar/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zijun Tian Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

None

Participants:

 Description   

This happened twice on a secondary node in the last week and caused the MongoDB down. Searched the issues and cannot find similar issues. Logs can be provided.

Can anyone help with this? Thanks! 
Sat Feb 19 08:37:22.644 E STORAGE [conn959063] WiredTiger error (5) [1645288642:644562][50690:0x7fec9d1c8700], file:collection-60341-8917267969775698840.wt, WT_CURSOR.next: __posix_file_read, 419: /mnt/mongodb/data/collection-60341-8917267969775698840.wt: handle-read: pread: failed to read 24576 bytes at offset 644296704: Input/output error Raw: [1645288642:644562][50690:0x7fec9d1c8700], file:collection-60341-8917267969775698840.wt, WT_CURSOR.next: __posix_file_read, 419: /mnt/mongodb/data/collection-60341-8917267969775698840.wt: handle-read: pread: failed to read 24576 bytes at offset 644296704: Input/output error
Sat Feb 19 08:37:22.644 F - [conn959063] Invariant failure: advanceRet resulted in status UnknownError: 5: Input/output error at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 1946
Sat Feb 19 08:37:22.645 F - [conn959063] \n\n***aborting after invariant() failure\n\n



 Comments   
Comment by Edwin Zhou [ 17/Mar/22 ]

Thank you for following up zijun.tian@tusimple.ai, I'll now close this issue.

If this issue occurs again, please attach the mongod.log files leading up to the failure.

Best,
Edwin

Comment by Zijun Tian [ 17/Mar/22 ]

Hi Edwin,

I believe the error was due to the bad disk volume, we have replaced it, thanks!

Comment by Edwin Zhou [ 17/Mar/22 ]

Hi zijun.tian@tusimple.ai,

Have you been able to look at your syslog or dmesg to investigate any storage layer or i/o errors? If this is still an issue for you, would you please also attach log files leading to this failure?

Best,
Edwin

Comment by Dmitry Agranat [ 23/Feb/22 ]

Hi zijun.tian@tusimple.ai, the reported "Input/output error" usually indicates an HW/OS issue. I suggest reviewing syslog and dmesg during the time of the reported event for any i/o errors.

Generated at Thu Feb 08 05:58:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.