[SERVER-23396] wiredTiger STORAGE Assertion Created: 29/Mar/16  Updated: 29/Mar/16  Resolved: 29/Mar/16

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: joe piscitella Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

3.5.0-51-generic #77~precise1-Ubuntu SMP Thu Jun 5 00:48:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux


Attachments: HTML File log2    
Operating System: ALL
Participants:

 Description   

2016-03-29T07:14:33.209-0700 E STORAGE  [WiredTigerRecordStoreThread for local.oplog.rs] WiredTiger (0) [1459260873:209671][18313:0x7fa3392bb700], file:collection-6--6088918010050475399.wt, cursor.next: read checksum error for 8192B block at offset 7092989952: calculated block checksum of 2027459797 doesn't match expected checksum of 3226078274
2016-03-29T07:14:33.209-0700 E STORAGE  [WiredTigerRecordStoreThread for local.oplog.rs] WiredTiger (0) [1459260873:209757][18313:0x7fa3392bb700], file:collection-6--6088918010050475399.wt, cursor.next: collection-6--6088918010050475399.wt: encountered an illegal file format or internal value
2016-03-29T07:14:33.209-0700 E STORAGE  [WiredTigerRecordStoreThread for local.oplog.rs] WiredTiger (-31804) [1459260873:209781][18313:0x7fa3392bb700], file:collection-6--6088918010050475399.wt, cursor.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-03-29T07:14:33.209-0700 I -        [WiredTigerRecordStoreThread for local.oplog.rs] Fatal Assertion 28558
2016-03-29T07:14:33.223-0700 I CONTROL  [WiredTigerRecordStoreThread for local.oplog.rs]
 0xf84362 0xf2b919 0xf0eb26 0xda5c81 0x13dde7c 0x13de02d 0x13de4a4 0x132a0c4 0x134534d 0x134a2c8 0x1346ea3 0x135c14e 0x135cb52 0x132e60a 0x137b90c 0xd931ad 0xd993e6 0xf1198c 0xfd386b 0x7fa33daa8e9a 0x7fa33cbb73fd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B84362","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"B2B919","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"B0EB26","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"9A5C81"},{"b":"400000","o":"FDDE7C","s":"__wt_eventv"},{"b":"400000","o":"FDE02D","s":"__wt_err"},{"b":"400000","o":"FDE4A4","s":"__wt_panic"},{"b":"400000","o":"F2A0C4","s":"__wt_bm_read"},{"b":"400000","o":"F4534D","s":"__wt_bt_read"},{"b":"400000","o":"F4A2C8","s":"__wt_cache_read"},{"b":"400000","o":"F46EA3","s":"__wt_page_in_func"},{"b":"400000","o":"F5C14E"},{"b":"400000","o":"F5CB52","s":"__wt_tree_walk"},{"b":"400000","o":"F2E60A","s":"__wt_btcur_next"},{"b":"400000","o":"F7B90C"},{"b":"400000","o":"9931AD","s":"_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE"},{"b":"400000","o":"9993E6"},{"b":"400000","o":"B1198C","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"400000","o":"BD386B"},{"b":"7FA33DAA1000","o":"7E9A"},{"b":"7FA33CAC3000","o":"F43FD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.0.10", "gitVersion" : "1e0512f8453d103987f5fbfb87b71e9a131c2a60", "uname" : { "sysname" : "Linux", "release" : "3.5.0-51-generic", "version" : "#77~precise1-Ubuntu SMP Thu Jun 5 00:48:28 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF0FC26000", "elfType" : 3 }, { "b" : "7FA33DAA1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FA33D899000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FA33D695000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FA33D395000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7FA33D099000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FA33CE83000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7FA33CAC3000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FA33DCBE000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf84362]
 mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf2b919]
 mongod(_ZN5mongo13fassertFailedEi+0x66) [0xf0eb26]
 mongod(+0x9A5C81) [0xda5c81]
 mongod(__wt_eventv+0x49C) [0x13dde7c]
 mongod(__wt_err+0x8D) [0x13de02d]
 mongod(__wt_panic+0x24) [0x13de4a4]
 mongod(__wt_bm_read+0x74) [0x132a0c4]
 mongod(__wt_bt_read+0x7D) [0x134534d]
 mongod(__wt_cache_read+0x98) [0x134a2c8]
 mongod(__wt_page_in_func+0x203) [0x1346ea3]
 mongod(+0xF5C14E) [0x135c14e]
 mongod(__wt_tree_walk+0x342) [0x135cb52]
 mongod(__wt_btcur_next+0x4BA) [0x132e60a]
 mongod(+0xF7B90C) [0x137b90c]
 mongod(_ZN5mongo21WiredTigerRecordStore27cappedDeleteAsNeeded_inlockEPNS_16OperationContextERKNS_8RecordIdE+0x2ED) [0xd931ad]
 mongod(+0x9993E6) [0xd993e6]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x12C) [0xf1198c]
 mongod(+0xBD386B) [0xfd386b]
 libpthread.so.0(+0x7E9A) [0x7fa33daa8e9a]
 libc.so.6(clone+0x6D) [0x7fa33cbb73fd]
-----  END BACKTRACE  -----
2016-03-29T07:14:33.223-0700 I -        [WiredTigerRecordStoreThread for local.oplog.rs]



 Comments   
Comment by Ramon Fernandez Marina [ 29/Mar/16 ]

Thanks for uploading the log jpiscitella. The errors you were seeing indicate that data in the collection-6--6088918010050475399.wt somehow got corrupted. The most common culprit is errors at the storage layer, so I'd recommend you search your system logs for storage-related error messages and check the health of your storage layer to make sure this doesn't happen again.

Since you've rebuilt this node there's not much we can do in terms of troubleshooting, so I'm going to close this ticket. If this happens again it would be helpful if you could save a copy of the data in your dbpath for further investigation.

Regards,
Ramón.

Comment by joe piscitella [ 29/Mar/16 ]

mongod terminated and wouldn't restart. since this was a member of a replica set I just rebuilt from the primary. the environment was our test environment that we updated from 3.0.6 to 3.0.10 this weekend.

log2 file attached as requested

Comment by Ramon Fernandez Marina [ 29/Mar/16 ]

jpiscitella, I've moved this ticket for the SERVER project, since you've run into this issue using MongoDB and not standalone WiredTiger.

Do you have the full logs since the last restart until you see this error in the logs? I'm looking for more details, specially what happens after the "END BACKTRACE" message. Does this node continue to operate or does it terminate?

Thanks,
Ramón.

Generated at Thu Feb 08 04:03:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.