[SERVER-17141] session->truncate failed with File Not Found in WiredTigerRecordStoreThread for local.oplog.rs Created: 31/Jan/15 Updated: 23/Sep/19 Resolved: 24/Feb/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage, WiredTiger |
| Affects Version/s: | 3.0.0-rc7 |
| Fix Version/s: | 3.0.0-rc10, 3.1.0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Michael Grundy | Assignee: | Keith Bostic (Inactive) |
| Resolution: | Done | Votes: | 1 |
| Labels: | 28qa, todo_in_code | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Backport Completed: | |||||||||||||||||||||
| Steps To Reproduce: | Using an r3.4xlarge instance (16 cpu, 122g ram, ephemeral storage for data) the attached mongo.conf (modded for dbpath and logpath), [https://github.com/tmcallaghan/iibench-mongodb|iibench] with the attached changes and shell script, and mongod run as follows: numactl --interleave=all ./mongodb-linux-x86_64-3.0.0-rc8/bin/mongod --config mongo.conf --storageEngine wiredTiger --wiredTigerCacheSizeGB 70 --master Workload is run on same machine. bash run.2.simple.bash 1000000 jdk-7u25, mongo-java-driver-2.12.5.jar. |
||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Running insert workload, primary failed about 12 hours in.
|
| Comments |
| Comment by Githook User [ 19/Feb/15 ] | ||||||||||||||||||||
|
Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}Message: (cherry picked from commit 4ca4224dae91ec067a38cd552b3b8c50999d7707) | ||||||||||||||||||||
| Comment by Githook User [ 19/Feb/15 ] | ||||||||||||||||||||
|
Author: {u'username': u'monkey101', u'name': u'Dan Pasette', u'email': u'dan@10gen.com'}Message: | ||||||||||||||||||||
| Comment by Githook User [ 17/Feb/15 ] | ||||||||||||||||||||
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: (cherry picked from commit d47529bb1ad51e65f5262ace4efcaff96062e028) | ||||||||||||||||||||
| Comment by Githook User [ 14/Feb/15 ] | ||||||||||||||||||||
|
Author: {u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}Message: | ||||||||||||||||||||
| Comment by Michael Grundy [ 12/Feb/15 ] | ||||||||||||||||||||
|
I'm able to reproduce it now, takes about 1800 seconds longer. BT from rc8 below.
| ||||||||||||||||||||
| Comment by Mark Callaghan [ 10/Feb/15 ] | ||||||||||||||||||||
|
Crash repeated after 3400 seconds. zlib still OK after 7000 seconds | ||||||||||||||||||||
| Comment by Mark Callaghan [ 10/Feb/15 ] | ||||||||||||||||||||
|
Put jmongoiibench.java in the "src" directory | ||||||||||||||||||||
| Comment by Michael Grundy [ 10/Feb/15 ] | ||||||||||||||||||||
|
Hi Mark - Could you put in your full mongod options, and the iibench option changes compared to the run.simple.bash script. I want to make sure I don't assume anything while I try to repro this. Thanks! | ||||||||||||||||||||
| Comment by Mark Callaghan [ 10/Feb/15 ] | ||||||||||||||||||||
|
I get this in rc8 after about 15 minutes. The workload is iibench-mongodb with 10 loader threads each doing 1 document/insert, 0 query threads and a 2000MB oplog. Let me know if you want more details. This occurs with snappy compression. The test with zlib compression has made it almost to 30 minutes without a crash.
| ||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 31/Jan/15 ] | ||||||||||||||||||||
|
michael.grundy@10gen.com, do you have any other system info to help track this?
|