Githook User added a comment - Jul 13 2015 04:10:36 AM +00:00 Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: WT-1962 Fix error handling of hotbackup unlock. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/bfea5498e41c43453f3a13203d1b3a78e95d41f2

Githook User added a comment - Jul 13 2015 04:10:37 AM +00:00 Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2054 from wiredtiger/backup-rwlock WT-1962 Make the hot_backup_lock a read/write lock. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/3866fa663b7244fec7fdb3930885ee5314736c45

Sue LoVerso added a comment - Sep 09 2015 08:23:16 PM +00:00 After adding a bunch of timing calls and other printfs to the code, here's a distilled sequence of events that causes the update-checkpoint-lsm.wtperf configuration to readily fail with a > 2 second latency. NOTE: This one is specific to LSM. I'm also able to cause a failure with update-checkpoint-btree.wtperf for long latency. I haven't gone through that output yet. The left column represents time in seconds. 53.7 CKPT Full checkpoint started Starts to get locks and call ckpt_list 53.8 LSM-worker checkpoints chunk 4. Final fsync takes 321 ms with SCHEMA 53.9 CKPT Checkpoint wt_checkpoint_list completes. Write leaf pages. 54.4 LSM acquires SCHEMA to switch to chunk 7. 54.7 Different LSM worker flushing chunk 5 waits for SCHEMA. 55.2 LSM switch, fsync of chunk 7 takes 637 ms 55.9 CKPT Calls checkpoint_sync w/o SCHEMA. 56.3 CKPT flushes chunk 5. Fsync takes 328 ms (initial flush w/o SCHEMA) 56.3 CKPT waits to acquire SCHEMA. 56.49 LSM switch releases SCHEMA, held 2036 ms. 56.49 CKPT acquires SCHEMA 56.49 USER thread enters LSM cursor update. waits to get SCHEMA. 56.49 CKPT calls __wt_checkpoint 56.8 CKPT calls __wt_checkpoint_sync 58.1 CKPT Fsync of chunk 6 takes 1315 ms 58.3 CKPT Fsync of chunk 5 takes 193 ms 58.3 CKPT releases SCHEMA 58.3 USER LSM cursor update took 1860 ms to get SCHEMA 58.3 CKPT Completes in 4634 ms 58.3 LSM Took 3595 ms to acquire SCHEMA 58.3 LSM worker completes checkpoint of chunk 5. 58.3 USER thread detects > 2 second latency

David Hows added a comment - Sep 24 2015 12:01:56 PM +00:00 Hey Sue LoVerso , Whats the status of this one? I've just seen a latency failure on update-checkpoint-btree.wtperf , which is the btree checkpoint task. http://build.wiredtiger.com:8080/job/wiredtiger-perf-checkpoint/22/console Could this issue still be hanging about?

Sue LoVerso added a comment - Sep 24 2015 02:19:06 PM +00:00 Yes, both of these issues still exist. I have not been able to repro the btree failure after adding timing debugging to understand where the time is being spent unfortunately. I have been able to get other btree tests to fail (all 500m-btree insert/update tests) with the instrumentation so hopefully WT-2124 and this one (for btree) have a common cause.