The voxer-130k-short wtperf run dropped between 10-15% in performance between builds  (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/287/) and  (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/288/) (here's the plot, it's the drop at the [end of July] (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/plot/).
The change that's responsible is number 4 in build 288 (839742b, "Switch from using the checkpoint lock to serialize bulk-load close and database checkpoints, to using a new WT_DATA_HANDLE lock").
Here are the runs:
I've been looking at this, and I'm suspicious there's an open/close pattern in LSM that I don't understand, where the close handle spinlock is being held by checkpoint (implying a bulk-loaded file, I think), and is blocking LSM from making forward progress? It could be the reverse, but that should just slow down a checkpoint, it shouldn't affect the read performance of the run.
@michaelcahill, any thoughts?