- 
    Type:
Bug
 - 
    Resolution: Done
 - 
    Priority:
Major - P3
 - 
    Affects Version/s: 2.8.0-rc5
 - 
    Component/s: Storage
 
- 
        Fully Compatible
 - 
        ALL
 - 
        None
 
- 
        None
 - 
        None
 - 
        None
 - 
        None
 - 
        None
 - 
        None
 
During one 10-minute run of a heavy mixed workload a period of several seconds of very low throughput was seen:

- During the pause all statistics appear to be consistent with a large number of cache evictions, resulting in a lot of writes, system cpu time, and context switches. This heavy eviction load seems like a plausible cause for the low throughput during this period.
 - Immediately before the pause the number of bytes in the cache jumped by about 4 GiB between two samples about 0.1 seconds apart. This is a plausible trigger for the heavy eviction load.
 - During the pause bytes and pages in cache declined slowly, presumably a result of the page evictions.
 - The pause ends with a sudden drop in the size of the cache by about 4 GiB between two samples about 0.1 seconds apart.
 
Here are the specific samples
- just before increase
"localTime":"2015-01-15T18:42:20.344Z"
"bytes currently in the cache":4315898330
"pages currently held in the cache":60756 
- just after increase
"localTime":"2015-01-15T18:42:20.472Z"
"bytes currently in the cache":8594367123
"pages currently held in the cache":60598 
- just before decrease
"localTime":"2015-01-15T18:42:25.058Z"
"bytes currently in the cache":7242022268
"pages currently held in the cache":35169 
- just after decrease
"localTime":"2015-01-15T18:42:25.170Z"
"bytes currently in the cache":2941126278
"pages currently held in the cache":35137 
The size of the sudden jump was: 7242022268 - 2941126278 = 3.98 GiB
The size of the sudden drop was: 8594367123 - 4315898330 = 4.01 GiB
Neither a sudden increase nor a sudden decrease of cache size by 4 GiB in 0.1 seconds due to real cached data is plausible.