Mark notes:

> I am running performance tests on WT 1.2.2 to revisit the rate at which it can perform disk reads for a read-only workload. I use a database with 10B rows and 250G bytes on disk. When the WT cache is small then performance is stable. When the WT cache is large then performance falls once eviction starts. I think you did a few changes previously to improve performance but I am curious about this. What in WT makes the eviction overhead a function of the cache size?

> Upgrading to a modern version of gcc + glib from older versions (gcc 4.4 -> 4.6, glibc 2.5 -> 2.13) fixed this for me. The call stack when it was slow is listed below: qsort called malloc. Eventually you might need your own sort function to avoid the risk from problems like that.
> sched_yield, __evict_get_page, __wt_evict_lru_page, *wt_page_in_func, *wt_row_search, *wt_btcur_search, *curfile_search,query_thread
> _int_malloc, malloc, qsort, *evict_lru_sort, *evict_lru, __evict_worker, __wt_cache_evict_server

Michael notes he's seen slow-down based on sorting in the eviction thread too.

We might want to consider not sorting at all in the eviction table, that is, rather than collecting a large set of pages and sorting them, keep a min/max value for a small number of pages and and/or maintain the smaller list in sorted order. The question is if maintaining a smaller pool of potential eviction candidates would affect our LRU-like behavior.

There are a number of other calls to qsort in the system, but none of them are currently sorting large numbers of items.