The voxer-130k-short wtperf run dropped between 10-15% in performance between builds [287] (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/287/) and [288] (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/288/) (here's the plot, it's the drop at the [end of July] (http://mjc.homeunix.org:8180/job/wiredtiger-perf-voxer-130k/plot/).

The change that's responsible is number 4 in build 288 (839742b, "Switch from using the checkpoint lock to serialize bulk-load close and database checkpoints, to using a new WT_DATA_HANDLE lock").

Here are the runs:

build 287:
read: avg: 21239788
insert: update: avg: 317920

build 287 + change 1
read: avg: 21346948
insert: update: avg: 320332

build 287 + change 4 (skipping changes 2 & 3, that was a change and its reversion)
read: avg: 18351485
insert: update: avg: 314058

I've been looking at this, and I'm suspicious there's an open/close pattern in LSM that I don't understand, where the close handle spinlock is being held by checkpoint (implying a bulk-loaded file, I think), and is blocking LSM from making forward progress? It could be the reverse, but that should just slow down a checkpoint, it shouldn't affect the read performance of the run.

@michaelcahill, any thoughts?