[SERVER-1558] Documents should write checksum on write, verify checksum on read Created: 03/Aug/10 Updated: 28/Oct/15 Resolved: 14/Feb/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 2.8.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Morgan Tocker | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 17 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
All platforms |
||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Description |
|
In InnoDB, crc page checksums are written as dirty pages are flushed from the buffer pool. On read, these checksums are verified. If the checksum fails, the server crashes awaiting intervention (startup with --innodb-force-recovery=N). At least for InnoDB, the cost is very little, since it takes microseconds around an operation that takes about milliseconds. The storage cost is also minimal. This is very helpful in failure scenarios. Check/repair in mongodb can currently only detect meta data corruption! |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 14/Feb/15 ] |
|
The WiredTiger storage engine as of v3.0 is configured by default to checksum blocks not otherwise protected by compression. This mode is called "uncompressed" |
| Comment by Matthias Götzke [ 07/Aug/10 ] |
|
Drive issues are even more important for us than sudden crashes (those can at least be detected with a dirty flag etc). Silent corruption due to filesystem errors are more subtle. Our archiving solution archives data and documents and all documents are tested regularily for corruption (and if necessary restored and the raid marked). Typically the filesystem and the raids should take care of that but sometimes they don't and we need to be able to have another barrier there. If we had an automated checksum within mongodb we might even move our documents to mongo (we would have to test sharding perfomance first though). |
| Comment by Eliot Horowitz (Inactive) [ 07/Aug/10 ] |
|
Single server durability in 1.8 will address that, but checksums might still be nice at some point for detecting drive errors, etc... |
| Comment by Morgan Tocker [ 07/Aug/10 ] |
|
True - the servers crashing itself part is optional. In Percona Server with XtraDB (alternative MySQL distribution) there is an enhancement to relax this consistency so that if one table is corrupt, the server stays up. In the case of mongoDB it could try and retrieve those documents from another replica. What I am most concerned about currently, is that if I check/repair a mongoDB database there are a lot of silent corruptions it can not tell me about provided they don't fall on a meta data boundary. For example: If inside a document I have long key which is just a sequence of A, i.e. 'AAAAAAA...'. and perform an operation to change it to all 'BBBBBB....' and the server crashes during the operation, I could have 1/2 A and 1/2 B. There will be no way to detect this corruption without a log or checksum. |
| Comment by Matthias Götzke [ 07/Aug/10 ] |
|
The server should not crash though. That could be an option, it should rather report the corruption (e.g. via driver level). A corruption of a small number of documents should not disallow accessing the other entries, since in this scenario they are individually checked anyway. |