random-rw: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 ... random-rw: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=8 fio-2.1.4 Starting 32 processes random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: Laying out IO file(s) (1 file(s) / 2048MB) random-rw: (groupid=0, jobs=1): err= 0: pid=28062: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=420865B/s, iops=102, runt=2552875msec slat (usec): min=216, max=204866, avg=9462.77, stdev=26298.90 clat (usec): min=6, max=6175.5K, avg=34118.47, stdev=70543.40 lat (usec): min=386, max=6183.5K, avg=43583.53, stdev=73787.62 clat percentiles (usec): | 1.00th=[ 1320], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[181248], 99.95th=[197632], | 99.99th=[2605056] bw (KB /s): min= 3, max= 1282, per=3.29%, avg=418.32, stdev=74.09 write: io=1023.4MB, bw=420336B/s, iops=102, runt=2552875msec slat (usec): min=6, max=6155.8K, avg=251.31, stdev=30945.38 clat (usec): min=87, max=6178.5K, avg=34055.35, stdev=78769.86 lat (usec): min=95, max=6178.6K, avg=34308.21, stdev=84644.54 clat percentiles (usec): | 1.00th=[ 1384], 5.00th=[ 3760], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[193536], | 99.99th=[5079040] bw (KB /s): min= 1, max= 1124, per=3.29%, avg=417.61, stdev=106.19 lat (usec) : 10=0.01%, 100=0.14%, 250=0.64%, 500=0.03%, 750=0.04% lat (usec) : 1000=0.04% lat (msec) : 2=0.51%, 4=4.22%, 10=22.68%, 20=46.69%, 50=8.59% lat (msec) : 100=0.39%, 250=15.99%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267122, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28063: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=413158B/s, iops=100, runt=2600496msec slat (usec): min=206, max=294266, avg=9462.61, stdev=26321.45 clat (usec): min=46, max=6337.9K, avg=34773.36, stdev=92142.36 lat (usec): min=390, max=6342.4K, avg=44238.32, stdev=94618.63 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[342016], | 99.99th=[5079040] bw (KB /s): min= 4, max= 1282, per=3.28%, avg=416.86, stdev=76.85 write: io=1023.4MB, bw=412638B/s, iops=100, runt=2600496msec slat (usec): min=6, max=6326.2K, avg=432.71, stdev=40263.53 clat (usec): min=90, max=6339.4K, avg=34672.18, stdev=85688.00 lat (usec): min=98, max=6339.4K, avg=35106.50, stdev=94670.71 clat percentiles (usec): | 1.00th=[ 1464], 5.00th=[ 3792], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[329728], | 99.99th=[4882432] bw (KB /s): min= 5, max= 1139, per=3.28%, avg=416.41, stdev=107.09 lat (usec) : 50=0.01%, 100=0.09%, 250=0.69%, 500=0.03%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.45%, 4=4.23%, 10=22.64%, 20=46.90%, 50=8.45% lat (msec) : 100=0.37%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=267605, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28064: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=412454B/s, iops=100, runt=2604936msec slat (usec): min=139, max=294800, avg=9455.48, stdev=26308.42 clat (usec): min=19, max=6444.7K, avg=34958.23, stdev=92914.39 lat (usec): min=333, max=6450.4K, avg=44416.04, stdev=95447.74 clat percentiles (usec): | 1.00th=[ 1304], 5.00th=[ 3696], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[552960], | 99.99th=[4882432] bw (KB /s): min= 4, max= 1434, per=3.28%, avg=416.99, stdev=77.74 write: io=1023.4MB, bw=411935B/s, iops=100, runt=2604936msec slat (usec): min=6, max=6302.9K, avg=457.04, stdev=40599.62 clat (usec): min=89, max=6324.5K, avg=34605.57, stdev=86151.61 lat (usec): min=97, max=6444.7K, avg=35064.21, stdev=95294.52 clat percentiles (usec): | 1.00th=[ 1416], 5.00th=[ 3792], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[329728], | 99.99th=[4423680] bw (KB /s): min= 1, max= 1250, per=3.28%, avg=416.36, stdev=107.45 lat (usec) : 20=0.01%, 100=0.13%, 250=0.67%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.50%, 4=4.24%, 10=22.66%, 20=46.76%, 50=8.52% lat (msec) : 100=0.39%, 250=15.97%, 500=0.01%, 750=0.01%, 2000=0.01% lat (msec) : >=2000=0.04% cpu : usr=0.09%, sys=0.51%, ctx=268006, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28065: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407441B/s, iops=99, runt=2636986msec slat (usec): min=133, max=203463, avg=9320.58, stdev=26171.58 clat (usec): min=7, max=6076.7K, avg=35358.42, stdev=107251.32 lat (usec): min=339, max=6080.6K, avg=44681.35, stdev=109447.62 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 3088], 10.00th=[ 5728], 20.00th=[ 8032], | 30.00th=[10048], 40.00th=[11840], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[407552], 99.95th=[2244608], | 99.99th=[5013504] bw (KB /s): min= 0, max= 3481, per=3.29%, avg=418.63, stdev=136.23 write: io=1023.4MB, bw=406928B/s, iops=99, runt=2636986msec slat (usec): min=6, max=5912.7K, avg=714.27, stdev=48435.46 clat (usec): min=88, max=6070.8K, avg=35061.22, stdev=97068.91 lat (usec): min=96, max=6070.9K, avg=35777.18, stdev=108572.69 clat percentiles (usec): | 1.00th=[ 636], 5.00th=[ 3152], 10.00th=[ 5664], 20.00th=[ 7968], | 30.00th=[10048], 40.00th=[11712], 50.00th=[13504], 60.00th=[15296], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[242688], 99.95th=[2007040], | 99.99th=[4685824] bw (KB /s): min= 0, max= 3274, per=3.29%, avg=418.22, stdev=159.62 lat (usec) : 10=0.01%, 100=0.13%, 250=0.65%, 500=0.11%, 750=0.23% lat (usec) : 1000=0.35% lat (msec) : 2=1.44%, 4=4.38%, 10=22.37%, 20=45.77%, 50=8.34% lat (msec) : 100=0.40%, 250=15.72%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.06% cpu : usr=0.09%, sys=0.52%, ctx=269371, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28066: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=414773B/s, iops=101, runt=2590370msec slat (usec): min=199, max=293989, avg=9449.28, stdev=26304.82 clat (usec): min=6, max=6326.7K, avg=34670.22, stdev=89777.28 lat (usec): min=386, max=6328.1K, avg=44121.86, stdev=92326.77 clat percentiles (usec): | 1.00th=[ 1320], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[296960], | 99.99th=[4947968] bw (KB /s): min= 3, max= 1436, per=3.28%, avg=417.52, stdev=76.20 write: io=1023.4MB, bw=414252B/s, iops=101, runt=2590370msec slat (usec): min=6, max=6297.6K, avg=407.92, stdev=39424.25 clat (usec): min=87, max=5477.7K, avg=34504.65, stdev=85450.76 lat (usec): min=96, max=6311.5K, avg=34914.15, stdev=94172.99 clat percentiles (usec): | 1.00th=[ 1400], 5.00th=[ 3792], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[346112], | 99.99th=[4882432] bw (KB /s): min= 3, max= 1252, per=3.28%, avg=417.07, stdev=105.95 lat (usec) : 10=0.01%, 100=0.13%, 250=0.65%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.48%, 4=4.19%, 10=22.71%, 20=46.91%, 50=8.41% lat (msec) : 100=0.36%, 250=15.99%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=267585, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28067: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407126B/s, iops=99, runt=2639024msec slat (usec): min=132, max=205141, avg=9139.65, stdev=25971.80 clat (usec): min=6, max=5678.2K, avg=35307.48, stdev=112644.80 lat (usec): min=272, max=5681.3K, avg=44449.61, stdev=114764.26 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 2288], 10.00th=[ 4576], 20.00th=[ 7584], | 30.00th=[ 9792], 40.00th=[11584], 50.00th=[13376], 60.00th=[15168], | 70.00th=[17792], 80.00th=[22656], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[1400832], 99.95th=[2506752], | 99.99th=[4947968] bw (KB /s): min= 2, max=13792, per=3.34%, avg=425.46, stdev=322.82 write: io=1023.4MB, bw=406614B/s, iops=99, runt=2639024msec slat (usec): min=6, max=5558.7K, avg=903.38, stdev=53495.06 clat (usec): min=87, max=5681.3K, avg=35166.60, stdev=108268.73 lat (usec): min=96, max=5681.3K, avg=36071.59, stdev=120720.76 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 2384], 10.00th=[ 4576], 20.00th=[ 7520], | 30.00th=[ 9792], 40.00th=[11456], 50.00th=[13248], 60.00th=[15168], | 70.00th=[17792], 80.00th=[22656], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[1236992], 99.95th=[2408448], | 99.99th=[4882432] bw (KB /s): min= 0, max=15120, per=3.34%, avg=424.56, stdev=349.32 lat (usec) : 10=0.01%, 100=0.15%, 250=0.64%, 500=0.20%, 750=0.53% lat (usec) : 1000=0.81% lat (msec) : 2=2.21%, 4=4.46%, 10=22.10%, 20=45.06%, 50=7.90% lat (msec) : 100=0.36%, 250=15.45%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.05%, >=2000=0.07% cpu : usr=0.09%, sys=0.53%, ctx=270402, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28068: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=409066B/s, iops=99, runt=2626508msec slat (usec): min=214, max=204978, avg=9511.98, stdev=26335.24 clat (usec): min=7, max=5660.9K, avg=35258.30, stdev=94224.25 lat (usec): min=372, max=5663.3K, avg=44772.72, stdev=96672.80 clat percentiles (usec): | 1.00th=[ 1352], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[12096], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23936], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[189440], 99.95th=[1236992], | 99.99th=[4489216] bw (KB /s): min= 1, max= 1423, per=3.26%, avg=414.27, stdev=79.56 write: io=1023.4MB, bw=408552B/s, iops=99, runt=2626508msec slat (usec): min=6, max=5652.8K, avg=482.45, stdev=41028.61 clat (usec): min=89, max=5661.3K, avg=34881.62, stdev=85885.26 lat (usec): min=98, max=5667.1K, avg=35365.70, stdev=95158.56 clat percentiles (usec): | 1.00th=[ 1480], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23936], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[333824], | 99.99th=[4292608] bw (KB /s): min= 2, max= 1252, per=3.26%, avg=413.94, stdev=110.25 lat (usec) : 10=0.01%, 100=0.11%, 250=0.68%, 500=0.02%, 750=0.04% lat (usec) : 1000=0.05% lat (msec) : 2=0.43%, 4=4.20%, 10=22.54%, 20=46.52%, 50=8.86% lat (msec) : 100=0.48%, 250=16.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.04% cpu : usr=0.09%, sys=0.51%, ctx=268450, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28069: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=414934B/s, iops=101, runt=2589368msec slat (usec): min=198, max=203859, avg=9449.13, stdev=26298.78 clat (usec): min=6, max=6337.9K, avg=34482.69, stdev=83688.32 lat (usec): min=342, max=6343.1K, avg=43934.29, stdev=86402.71 clat percentiles (usec): | 1.00th=[ 1320], 5.00th=[ 3696], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[337920], | 99.99th=[4423680] bw (KB /s): min= 5, max= 1660, per=3.28%, avg=417.15, stdev=77.63 write: io=1023.4MB, bw=414412B/s, iops=101, runt=2589368msec slat (usec): min=6, max=6319.6K, avg=403.56, stdev=38315.10 clat (usec): min=87, max=6337.9K, avg=34665.78, stdev=87883.19 lat (usec): min=95, max=6337.9K, avg=35070.98, stdev=95891.69 clat percentiles (usec): | 1.00th=[ 1448], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[382976], | 99.99th=[4554752] bw (KB /s): min= 2, max= 1482, per=3.28%, avg=416.67, stdev=106.66 lat (usec) : 10=0.01%, 100=0.14%, 250=0.65%, 500=0.03%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.47%, 4=4.23%, 10=22.69%, 20=46.81%, 50=8.49% lat (msec) : 100=0.38%, 250=15.97%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=268124, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28070: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407321B/s, iops=99, runt=2637762msec slat (usec): min=136, max=294565, avg=9298.88, stdev=26153.93 clat (usec): min=6, max=6299.8K, avg=35111.48, stdev=100712.32 lat (usec): min=273, max=6302.4K, avg=44412.79, stdev=103101.55 clat percentiles (usec): | 1.00th=[ 620], 5.00th=[ 3024], 10.00th=[ 5664], 20.00th=[ 7968], | 30.00th=[10048], 40.00th=[11840], 50.00th=[13504], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[207872], 99.95th=[2211840], | 99.99th=[4882432] bw (KB /s): min= 2, max= 6208, per=3.31%, avg=420.86, stdev=170.78 write: io=1023.4MB, bw=406809B/s, iops=99, runt=2637762msec slat (usec): min=6, max=6231.3K, avg=738.75, stdev=50421.99 clat (usec): min=88, max=6297.6K, avg=35329.27, stdev=109670.35 lat (usec): min=96, max=6304.6K, avg=36069.66, stdev=120671.85 clat percentiles (usec): | 1.00th=[ 612], 5.00th=[ 3056], 10.00th=[ 5600], 20.00th=[ 7904], | 30.00th=[10048], 40.00th=[11712], 50.00th=[13504], 60.00th=[15296], | 70.00th=[17792], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[342016], 99.95th=[2670592], | 99.99th=[5079040] bw (KB /s): min= 3, max= 6272, per=3.31%, avg=420.31, stdev=192.51 lat (usec) : 10=0.01%, 100=0.13%, 250=0.66%, 500=0.12%, 750=0.27% lat (usec) : 1000=0.39% lat (msec) : 2=1.56%, 4=4.31%, 10=22.35%, 20=45.78%, 50=8.25% lat (msec) : 100=0.40%, 250=15.69%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.06% cpu : usr=0.09%, sys=0.52%, ctx=268945, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28071: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=422413B/s, iops=103, runt=2543523msec slat (usec): min=217, max=204173, avg=9481.96, stdev=26303.15 clat (usec): min=6, max=6232.9K, avg=34059.24, stdev=67013.17 lat (usec): min=398, max=6238.8K, avg=43543.73, stdev=70419.17 clat percentiles (usec): | 1.00th=[ 1304], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23936], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[181248], 99.95th=[189440], | 99.99th=[2867200] bw (KB /s): min= 7, max= 1660, per=3.28%, avg=417.58, stdev=75.51 write: io=1023.4MB, bw=421881B/s, iops=102, runt=2543523msec slat (usec): min=6, max=6220.2K, avg=195.49, stdev=25496.51 clat (usec): min=85, max=6238.8K, avg=33864.84, stdev=67341.35 lat (usec): min=94, max=6238.8K, avg=34061.96, stdev=72027.84 clat percentiles (usec): | 1.00th=[ 1400], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[181248], 99.95th=[189440], | 99.99th=[2867200] bw (KB /s): min= 5, max= 1482, per=3.28%, avg=416.96, stdev=107.08 lat (usec) : 10=0.01%, 100=0.12%, 250=0.66%, 500=0.03%, 750=0.04% lat (usec) : 1000=0.05% lat (msec) : 2=0.45%, 4=4.16%, 10=22.73%, 20=46.50%, 50=8.79% lat (msec) : 100=0.43%, 250=16.00%, 500=0.01%, 1000=0.01%, 2000=0.01% lat (msec) : >=2000=0.01% cpu : usr=0.09%, sys=0.51%, ctx=267622, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28072: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=417269B/s, iops=101, runt=2574875msec slat (usec): min=209, max=202601, avg=9432.89, stdev=26287.65 clat (usec): min=6, max=6327.3K, avg=34458.21, stdev=84165.45 lat (usec): min=363, max=6331.6K, avg=43893.52, stdev=86900.37 clat percentiles (usec): | 1.00th=[ 1272], 5.00th=[ 3696], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[226304], | 99.99th=[4751360] bw (KB /s): min= 2, max= 1381, per=3.29%, avg=418.61, stdev=76.63 write: io=1023.4MB, bw=416744B/s, iops=101, runt=2574875msec slat (usec): min=5, max=6306.5K, avg=364.57, stdev=37311.61 clat (usec): min=87, max=6327.3K, avg=34303.08, stdev=84188.99 lat (usec): min=96, max=6327.3K, avg=34669.27, stdev=92062.97 clat percentiles (usec): | 1.00th=[ 1400], 5.00th=[ 3760], 10.00th=[ 6112], 20.00th=[ 8256], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[181248], 99.95th=[228352], | 99.99th=[4882432] bw (KB /s): min= 2, max= 1261, per=3.29%, avg=418.04, stdev=106.54 lat (usec) : 10=0.01%, 100=0.13%, 250=0.66%, 500=0.03%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.53%, 4=4.24%, 10=22.73%, 20=46.78%, 50=8.44% lat (msec) : 100=0.35%, 250=15.99%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267977, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28073: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407232B/s, iops=99, runt=2638341msec slat (usec): min=133, max=294449, avg=9213.59, stdev=26063.44 clat (usec): min=6, max=5688.4K, avg=35411.46, stdev=112666.68 lat (usec): min=304, max=5692.5K, avg=44627.46, stdev=114797.46 clat percentiles (usec): | 1.00th=[ 580], 5.00th=[ 2736], 10.00th=[ 5280], 20.00th=[ 7776], | 30.00th=[ 9920], 40.00th=[11712], 50.00th=[13376], 60.00th=[15296], | 70.00th=[17792], 80.00th=[22656], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[358400], 99.95th=[2637824], | 99.99th=[5079040] bw (KB /s): min= 2, max=11984, per=3.33%, avg=423.70, stdev=248.71 write: io=1023.4MB, bw=406719B/s, iops=99, runt=2638341msec slat (usec): min=6, max=5557.2K, avg=826.50, stdev=52987.19 clat (usec): min=89, max=5574.7K, avg=35044.38, stdev=107178.08 lat (usec): min=98, max=5574.7K, avg=35872.54, stdev=119635.15 clat percentiles (usec): | 1.00th=[ 580], 5.00th=[ 2768], 10.00th=[ 5216], 20.00th=[ 7712], | 30.00th=[ 9920], 40.00th=[11584], 50.00th=[13376], 60.00th=[15168], | 70.00th=[17792], 80.00th=[22656], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[366592], 99.95th=[2506752], | 99.99th=[4882432] bw (KB /s): min= 2, max=12528, per=3.33%, avg=423.06, stdev=270.72 lat (usec) : 10=0.01%, 100=0.11%, 250=0.67%, 500=0.15%, 750=0.37% lat (usec) : 1000=0.59% lat (msec) : 2=1.91%, 4=4.32%, 10=22.42%, 20=45.52%, 50=7.88% lat (msec) : 100=0.37%, 250=15.57%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.06% cpu : usr=0.09%, sys=0.53%, ctx=269903, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28074: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=415081B/s, iops=101, runt=2588452msec slat (usec): min=206, max=294434, avg=9452.07, stdev=26309.34 clat (usec): min=7, max=6318.1K, avg=34687.45, stdev=92072.77 lat (usec): min=374, max=6323.7K, avg=44142.02, stdev=94599.90 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[226304], | 99.99th=[4882432] bw (KB /s): min= 0, max= 1405, per=3.29%, avg=417.93, stdev=74.77 write: io=1023.4MB, bw=414558B/s, iops=101, runt=2588452msec slat (usec): min=6, max=6304.3K, avg=396.68, stdev=39497.39 clat (usec): min=90, max=6321.5K, avg=34436.45, stdev=83316.76 lat (usec): min=99, max=6323.2K, avg=34834.81, stdev=92188.46 clat percentiles (usec): | 1.00th=[ 1448], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[181248], 99.95th=[203776], | 99.99th=[4685824] bw (KB /s): min= 3, max= 1221, per=3.29%, avg=417.63, stdev=106.79 lat (usec) : 10=0.01%, 100=0.11%, 250=0.67%, 500=0.03%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.47%, 4=4.17%, 10=22.75%, 20=46.83%, 50=8.48% lat (msec) : 100=0.37%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=268128, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28075: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=408210B/s, iops=99, runt=2632016msec slat (usec): min=137, max=295016, avg=9482.64, stdev=26319.28 clat (usec): min=6, max=6324.6K, avg=35076.57, stdev=92176.41 lat (usec): min=267, max=6329.8K, avg=44561.68, stdev=94685.08 clat percentiles (usec): | 1.00th=[ 1112], 5.00th=[ 3696], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23936], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[189440], 99.95th=[634880], | 99.99th=[4358144] bw (KB /s): min= 4, max= 1917, per=3.26%, avg=414.65, stdev=82.58 write: io=1023.4MB, bw=407697B/s, iops=99, runt=2632016msec slat (usec): min=6, max=6291.5K, avg=532.66, stdev=43782.87 clat (usec): min=89, max=6315.3K, avg=35210.68, stdev=96805.80 lat (usec): min=98, max=6315.3K, avg=35745.01, stdev=106299.24 clat percentiles (usec): | 1.00th=[ 1176], 5.00th=[ 3760], 10.00th=[ 6112], 20.00th=[ 8256], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23936], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[189440], 99.95th=[1056768], | 99.99th=[4816896] bw (KB /s): min= 1, max= 1970, per=3.26%, avg=414.26, stdev=112.38 lat (usec) : 10=0.01%, 100=0.08%, 250=0.70%, 500=0.04%, 750=0.05% lat (usec) : 1000=0.07% lat (msec) : 2=0.57%, 4=4.18%, 10=22.72%, 20=46.39%, 50=8.69% lat (msec) : 100=0.44%, 250=15.99%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.05% cpu : usr=0.09%, sys=0.52%, ctx=268638, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28076: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=419252B/s, iops=102, runt=2562696msec slat (usec): min=213, max=203064, avg=9476.54, stdev=26307.26 clat (usec): min=6, max=6198.8K, avg=34368.90, stdev=74599.81 lat (usec): min=415, max=6201.8K, avg=43847.91, stdev=77625.57 clat percentiles (usec): | 1.00th=[ 1448], 5.00th=[ 3792], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[203776], | 99.99th=[4423680] bw (KB /s): min= 5, max= 1412, per=3.28%, avg=417.39, stdev=73.94 write: io=1023.4MB, bw=418725B/s, iops=102, runt=2562696msec slat (usec): min=6, max=6189.1K, avg=274.11, stdev=31083.87 clat (usec): min=89, max=6207.5K, avg=34067.17, stdev=75380.93 lat (usec): min=98, max=6214.5K, avg=34342.96, stdev=81558.75 clat percentiles (usec): | 1.00th=[ 1592], 5.00th=[ 3856], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[181248], 99.95th=[193536], | 99.99th=[4292608] bw (KB /s): min= 5, max= 1234, per=3.28%, avg=416.90, stdev=105.46 lat (usec) : 10=0.01%, 100=0.10%, 250=0.68%, 500=0.03%, 750=0.02% lat (usec) : 1000=0.03% lat (msec) : 2=0.40%, 4=4.18%, 10=22.78%, 20=46.74%, 50=8.57% lat (msec) : 100=0.42%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267638, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28077: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407710B/s, iops=99, runt=2635248msec slat (usec): min=137, max=202420, avg=9408.23, stdev=26249.14 clat (usec): min=6, max=6285.1K, avg=35187.40, stdev=95976.27 lat (usec): min=254, max=6290.5K, avg=44598.09, stdev=98391.70 clat percentiles (usec): | 1.00th=[ 764], 5.00th=[ 3376], 10.00th=[ 5984], 20.00th=[ 8160], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[232448], 99.95th=[2056192], | 99.99th=[4489216] bw (KB /s): min= 7, max= 2883, per=3.27%, avg=416.02, stdev=112.10 write: io=1023.4MB, bw=407197B/s, iops=99, runt=2635248msec slat (usec): min=6, max=6262.7K, avg=620.06, stdev=44819.04 clat (usec): min=87, max=6290.5K, avg=35185.99, stdev=96343.05 lat (usec): min=96, max=6290.5K, avg=35807.63, stdev=106206.57 clat percentiles (usec): | 1.00th=[ 764], 5.00th=[ 3472], 10.00th=[ 5920], 20.00th=[ 8160], | 30.00th=[10176], 40.00th=[11840], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[207872], 99.95th=[2007040], | 99.99th=[4882432] bw (KB /s): min= 0, max= 2975, per=3.27%, avg=415.52, stdev=137.33 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.06%, 750=0.14% lat (usec) : 1000=0.21% lat (msec) : 2=0.98%, 4=4.27%, 10=22.46%, 20=46.14%, 50=8.56% lat (msec) : 100=0.43%, 250=15.85%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.05% cpu : usr=0.09%, sys=0.52%, ctx=268425, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28078: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=415626B/s, iops=101, runt=2585058msec slat (usec): min=196, max=294984, avg=9477.10, stdev=26322.67 clat (usec): min=7, max=5282.3K, avg=34556.48, stdev=81444.55 lat (usec): min=386, max=5285.9K, avg=44036.00, stdev=84235.34 clat percentiles (usec): | 1.00th=[ 1448], 5.00th=[ 3760], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[240640], | 99.99th=[4423680] bw (KB /s): min= 0, max= 1663, per=3.28%, avg=416.67, stdev=75.85 write: io=1023.4MB, bw=415103B/s, iops=101, runt=2585058msec slat (usec): min=6, max=5270.5K, avg=358.74, stdev=35378.20 clat (usec): min=91, max=5431.6K, avg=34477.01, stdev=81185.92 lat (usec): min=99, max=5438.2K, avg=34837.41, stdev=88563.91 clat percentiles (usec): | 1.00th=[ 1544], 5.00th=[ 3888], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[222208], | 99.99th=[4358144] bw (KB /s): min= 0, max= 1465, per=3.28%, avg=416.21, stdev=107.02 lat (usec) : 10=0.01%, 100=0.09%, 250=0.70%, 500=0.03%, 750=0.02% lat (usec) : 1000=0.04% lat (msec) : 2=0.41%, 4=4.15%, 10=22.74%, 20=46.82%, 50=8.54% lat (msec) : 100=0.39%, 250=16.04%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.52%, ctx=268037, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28079: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=418290B/s, iops=102, runt=2568593msec slat (usec): min=176, max=206432, avg=9476.86, stdev=26318.36 clat (usec): min=6, max=5438.5K, avg=34459.83, stdev=80051.22 lat (usec): min=401, max=5443.2K, avg=43939.22, stdev=82987.34 clat percentiles (usec): | 1.00th=[ 1320], 5.00th=[ 3760], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[207872], | 99.99th=[4292608] bw (KB /s): min= 2, max= 1414, per=3.28%, avg=417.17, stdev=74.23 write: io=1023.4MB, bw=417764B/s, iops=101, runt=2568593msec slat (usec): min=6, max=5428.3K, avg=296.38, stdev=32106.61 clat (usec): min=88, max=5440.4K, avg=34133.68, stdev=73315.84 lat (usec): min=97, max=5444.1K, avg=34431.69, stdev=80134.90 clat percentiles (usec): | 1.00th=[ 1416], 5.00th=[ 3856], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[203776], | 99.99th=[3686400] bw (KB /s): min= 0, max= 1242, per=3.28%, avg=416.75, stdev=105.36 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.44%, 4=4.13%, 10=22.69%, 20=46.81%, 50=8.57% lat (msec) : 100=0.42%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267791, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28080: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407398B/s, iops=99, runt=2637262msec slat (usec): min=128, max=293973, avg=9337.02, stdev=26173.61 clat (usec): min=6, max=6134.1K, avg=35401.68, stdev=105138.71 lat (usec): min=243, max=6137.4K, avg=44741.15, stdev=107373.89 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 3120], 10.00th=[ 5728], 20.00th=[ 8032], | 30.00th=[10048], 40.00th=[11840], 50.00th=[13504], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[552960], 99.95th=[2277376], | 99.99th=[5079040] bw (KB /s): min= 1, max= 4348, per=3.29%, avg=418.68, stdev=147.90 write: io=1023.4MB, bw=406886B/s, iops=99, runt=2637262msec slat (usec): min=6, max=6109.6K, avg=698.78, stdev=47251.62 clat (usec): min=86, max=6136.4K, avg=35025.43, stdev=95010.29 lat (usec): min=97, max=6139.7K, avg=35725.87, stdev=106086.77 clat percentiles (usec): | 1.00th=[ 636], 5.00th=[ 3184], 10.00th=[ 5664], 20.00th=[ 7968], | 30.00th=[10048], 40.00th=[11712], 50.00th=[13504], 60.00th=[15296], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[203776], 99.95th=[2007040], | 99.99th=[4292608] bw (KB /s): min= 2, max= 4268, per=3.29%, avg=418.20, stdev=170.06 lat (usec) : 10=0.01%, 100=0.11%, 250=0.67%, 500=0.10%, 750=0.23% lat (usec) : 1000=0.36% lat (msec) : 2=1.40%, 4=4.29%, 10=22.42%, 20=45.66%, 50=8.50% lat (msec) : 100=0.42%, 250=15.74%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.03%, >=2000=0.06% cpu : usr=0.09%, sys=0.53%, ctx=269494, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28081: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407542B/s, iops=99, runt=2636332msec slat (usec): min=131, max=203555, avg=9383.52, stdev=26210.56 clat (usec): min=7, max=6146.2K, avg=35232.55, stdev=98087.45 lat (usec): min=292, max=6151.5K, avg=44618.59, stdev=100464.08 clat percentiles (usec): | 1.00th=[ 724], 5.00th=[ 3312], 10.00th=[ 5920], 20.00th=[ 8160], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[205824], 99.95th=[2007040], | 99.99th=[4816896] bw (KB /s): min= 2, max= 3224, per=3.28%, avg=417.00, stdev=123.54 write: io=1023.4MB, bw=407029B/s, iops=99, runt=2636332msec slat (usec): min=6, max=6127.6K, avg=648.18, stdev=45866.48 clat (usec): min=91, max=6146.3K, avg=35169.98, stdev=97402.90 lat (usec): min=100, max=6146.3K, avg=35819.84, stdev=107676.77 clat percentiles (usec): | 1.00th=[ 724], 5.00th=[ 3344], 10.00th=[ 5856], 20.00th=[ 8096], | 30.00th=[10176], 40.00th=[11840], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[203776], 99.95th=[2007040], | 99.99th=[4816896] bw (KB /s): min= 2, max= 3252, per=3.28%, avg=416.40, stdev=148.24 lat (usec) : 10=0.01%, 100=0.08%, 250=0.70%, 500=0.08%, 750=0.17% lat (usec) : 1000=0.24% lat (msec) : 2=1.12%, 4=4.31%, 10=22.30%, 20=46.00%, 50=8.66% lat (msec) : 100=0.45%, 250=15.81%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.05% cpu : usr=0.09%, sys=0.52%, ctx=269431, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28082: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=417438B/s, iops=101, runt=2573834msec slat (usec): min=144, max=206013, avg=9480.14, stdev=26331.09 clat (usec): min=6, max=5715.3K, avg=34534.76, stdev=81199.34 lat (usec): min=365, max=5718.8K, avg=44017.37, stdev=84030.77 clat percentiles (usec): | 1.00th=[ 1368], 5.00th=[ 3792], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[203776], | 99.99th=[4816896] bw (KB /s): min= 13, max= 1488, per=3.28%, avg=417.13, stdev=74.76 write: io=1023.4MB, bw=416913B/s, iops=101, runt=2573834msec slat (usec): min=6, max=5697.8K, avg=313.15, stdev=34288.67 clat (usec): min=89, max=5715.7K, avg=34198.55, stdev=78166.45 lat (usec): min=98, max=5715.7K, avg=34513.36, stdev=85360.30 clat percentiles (usec): | 1.00th=[ 1480], 5.00th=[ 3856], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[205824], | 99.99th=[4358144] bw (KB /s): min= 10, max= 1275, per=3.28%, avg=416.49, stdev=105.66 lat (usec) : 10=0.01%, 100=0.11%, 250=0.68%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.04% lat (msec) : 2=0.42%, 4=4.15%, 10=22.78%, 20=46.72%, 50=8.57% lat (msec) : 100=0.39%, 250=16.05%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267558, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28083: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=413322B/s, iops=100, runt=2599467msec slat (usec): min=187, max=294162, avg=9483.03, stdev=26327.78 clat (usec): min=7, max=5462.5K, avg=34827.37, stdev=87350.95 lat (usec): min=488, max=5466.8K, avg=44312.76, stdev=90025.60 clat percentiles (usec): | 1.00th=[ 1384], 5.00th=[ 3760], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[569344], | 99.99th=[4816896] bw (KB /s): min= 1, max= 1575, per=3.27%, avg=415.94, stdev=77.02 write: io=1023.4MB, bw=412802B/s, iops=100, runt=2599467msec slat (usec): min=6, max=5436.9K, avg=408.57, stdev=37401.81 clat (usec): min=87, max=5462.5K, avg=34590.47, stdev=81782.86 lat (usec): min=95, max=5462.5K, avg=35000.61, stdev=89932.19 clat percentiles (usec): | 1.00th=[ 1560], 5.00th=[ 3856], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[329728], | 99.99th=[4292608] bw (KB /s): min= 1, max= 1395, per=3.27%, avg=415.49, stdev=106.62 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.03% lat (msec) : 2=0.41%, 4=4.19%, 10=22.66%, 20=46.73%, 50=8.67% lat (msec) : 100=0.41%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=267646, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28084: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=408119B/s, iops=99, runt=2632607msec slat (usec): min=147, max=294909, avg=9473.53, stdev=26308.41 clat (usec): min=6, max=5606.1K, avg=35297.74, stdev=96166.61 lat (usec): min=256, max=5612.9K, avg=44773.84, stdev=98608.49 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 3664], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[189440], 99.95th=[1712128], | 99.99th=[4751360] bw (KB /s): min= 1, max= 2225, per=3.26%, avg=414.88, stdev=90.01 write: io=1023.4MB, bw=407605B/s, iops=99, runt=2632607msec slat (usec): min=6, max=5459.9K, avg=543.82, stdev=43555.22 clat (usec): min=89, max=5612.9K, avg=35005.14, stdev=92509.26 lat (usec): min=98, max=5612.9K, avg=35550.64, stdev=102313.14 clat percentiles (usec): | 1.00th=[ 1032], 5.00th=[ 3696], 10.00th=[ 6112], 20.00th=[ 8256], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[187392], 99.95th=[987136], | 99.99th=[4423680] bw (KB /s): min= 1, max= 2258, per=3.26%, avg=414.34, stdev=116.09 lat (usec) : 10=0.01%, 100=0.10%, 250=0.69%, 500=0.04%, 750=0.07% lat (usec) : 1000=0.10% lat (msec) : 2=0.66%, 4=4.12%, 10=22.64%, 20=46.45%, 50=8.65% lat (msec) : 100=0.46%, 250=15.95%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.04% cpu : usr=0.09%, sys=0.52%, ctx=268295, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28085: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=408183B/s, iops=99, runt=2632194msec slat (usec): min=146, max=335173, avg=9461.30, stdev=26304.06 clat (usec): min=6, max=6322.9K, avg=35256.34, stdev=99057.36 lat (usec): min=252, max=6325.7K, avg=44720.10, stdev=101372.95 clat percentiles (usec): | 1.00th=[ 1048], 5.00th=[ 3664], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[191488], 99.95th=[1515520], | 99.99th=[4947968] bw (KB /s): min= 3, max= 2160, per=3.26%, avg=415.32, stdev=84.34 write: io=1023.4MB, bw=407669B/s, iops=99, runt=2632194msec slat (usec): min=6, max=6305.5K, avg=554.48, stdev=45264.71 clat (usec): min=90, max=6314.9K, avg=35035.64, stdev=94733.68 lat (usec): min=98, max=6314.9K, avg=35591.80, stdev=105016.16 clat percentiles (usec): | 1.00th=[ 1064], 5.00th=[ 3728], 10.00th=[ 6048], 20.00th=[ 8256], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[189440], 99.95th=[428032], | 99.99th=[4947968] bw (KB /s): min= 2, max= 2246, per=3.27%, avg=414.90, stdev=113.30 lat (usec) : 10=0.01%, 100=0.08%, 250=0.70%, 500=0.04%, 750=0.06% lat (usec) : 1000=0.09% lat (msec) : 2=0.62%, 4=4.15%, 10=22.56%, 20=46.62%, 50=8.63% lat (msec) : 100=0.41%, 250=15.98%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.04% cpu : usr=0.09%, sys=0.52%, ctx=268331, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28086: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=418725B/s, iops=102, runt=2565924msec slat (usec): min=215, max=294710, avg=9471.43, stdev=26317.38 clat (usec): min=7, max=6323.2K, avg=34136.70, stdev=78135.43 lat (usec): min=399, max=6326.8K, avg=43610.58, stdev=81088.27 clat percentiles (usec): | 1.00th=[ 1320], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[181248], 99.95th=[199680], | 99.99th=[4292608] bw (KB /s): min= 5, max= 1688, per=3.28%, avg=417.76, stdev=75.86 write: io=1023.4MB, bw=418198B/s, iops=102, runt=2565924msec slat (usec): min=6, max=6297.6K, avg=291.90, stdev=34227.67 clat (usec): min=87, max=6323.3K, avg=34385.85, stdev=80868.89 lat (usec): min=96, max=6323.4K, avg=34679.36, stdev=87781.12 clat percentiles (usec): | 1.00th=[ 1416], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[207872], | 99.99th=[4816896] bw (KB /s): min= 5, max= 1487, per=3.28%, avg=417.37, stdev=107.25 lat (usec) : 10=0.01%, 100=0.13%, 250=0.66%, 500=0.03%, 750=0.04% lat (usec) : 1000=0.05% lat (msec) : 2=0.44%, 4=4.18%, 10=22.73%, 20=46.69%, 50=8.61% lat (msec) : 100=0.41%, 250=16.00%, 500=0.01%, 750=0.01%, 2000=0.01% lat (msec) : >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267293, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28087: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=417375B/s, iops=101, runt=2574223msec slat (usec): min=195, max=294344, avg=9459.92, stdev=26315.28 clat (usec): min=24, max=6107.5K, avg=34461.65, stdev=83898.16 lat (usec): min=341, max=6113.4K, avg=43923.97, stdev=86721.35 clat percentiles (usec): | 1.00th=[ 1304], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[230400], | 99.99th=[4620288] bw (KB /s): min= 2, max= 1591, per=3.28%, avg=417.89, stdev=76.20 write: io=1023.4MB, bw=416850B/s, iops=101, runt=2574223msec slat (usec): min=6, max=6084.5K, avg=335.04, stdev=35694.43 clat (usec): min=88, max=6106.5K, avg=34282.23, stdev=79953.75 lat (usec): min=96, max=6107.7K, avg=34618.93, stdev=87571.79 clat percentiles (usec): | 1.00th=[ 1400], 5.00th=[ 3792], 10.00th=[ 6112], 20.00th=[ 8256], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[183296], 99.95th=[199680], | 99.99th=[4620288] bw (KB /s): min= 2, max= 1404, per=3.28%, avg=417.38, stdev=106.90 lat (usec) : 50=0.01%, 100=0.12%, 250=0.66%, 500=0.03%, 750=0.03% lat (usec) : 1000=0.05% lat (msec) : 2=0.47%, 4=4.26%, 10=22.74%, 20=46.71%, 50=8.51% lat (msec) : 100=0.38%, 250=16.00%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=0.09%, sys=0.51%, ctx=267715, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28088: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=413154B/s, iops=100, runt=2600524msec slat (usec): min=211, max=292491, avg=9464.40, stdev=26306.66 clat (usec): min=6, max=6246.1K, avg=35004.91, stdev=91663.93 lat (usec): min=382, max=6256.6K, avg=44471.77, stdev=94195.92 clat percentiles (usec): | 1.00th=[ 1304], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[387072], | 99.99th=[4816896] bw (KB /s): min= 1, max= 1453, per=3.28%, avg=416.72, stdev=77.62 write: io=1023.4MB, bw=412634B/s, iops=100, runt=2600524msec slat (usec): min=6, max=6175.5K, avg=430.93, stdev=39277.24 clat (usec): min=87, max=6211.4K, avg=34440.98, stdev=83019.71 lat (usec): min=96, max=6244.6K, avg=34873.56, stdev=91873.63 clat percentiles (usec): | 1.00th=[ 1464], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[259072], | 99.99th=[4292608] bw (KB /s): min= 0, max= 1231, per=3.27%, avg=416.09, stdev=107.30 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.04% lat (msec) : 2=0.47%, 4=4.21%, 10=22.69%, 20=46.70%, 50=8.60% lat (msec) : 100=0.40%, 250=15.99%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=267779, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28089: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=420403B/s, iops=102, runt=2555681msec slat (usec): min=210, max=294170, avg=9488.40, stdev=26314.69 clat (usec): min=6, max=5665.4K, avg=34051.75, stdev=70737.65 lat (usec): min=380, max=5669.2K, avg=43542.54, stdev=73999.21 clat percentiles (usec): | 1.00th=[ 1400], 5.00th=[ 3760], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[12096], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[183296], 99.95th=[234496], | 99.99th=[3424256] bw (KB /s): min= 9, max= 1358, per=3.28%, avg=416.70, stdev=73.68 write: io=1023.4MB, bw=419874B/s, iops=102, runt=2555681msec slat (usec): min=6, max=5647.2K, avg=235.85, stdev=27401.75 clat (usec): min=89, max=5665.3K, avg=34197.20, stdev=68991.83 lat (usec): min=97, max=5665.4K, avg=34434.72, stdev=74242.11 clat percentiles (usec): | 1.00th=[ 1560], 5.00th=[ 3856], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[234496], | 99.99th=[3194880] bw (KB /s): min= 16, max= 1171, per=3.27%, avg=416.03, stdev=103.23 lat (usec) : 10=0.01%, 100=0.11%, 250=0.67%, 500=0.03%, 750=0.04% lat (usec) : 1000=0.04% lat (msec) : 2=0.42%, 4=4.17%, 10=22.61%, 20=46.70%, 50=8.73% lat (msec) : 100=0.47%, 250=15.97%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=0.09%, sys=0.51%, ctx=267453, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28090: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=414133B/s, iops=101, runt=2594375msec slat (usec): min=214, max=193840, avg=9482.87, stdev=26326.91 clat (usec): min=6, max=5853.8K, avg=34699.86, stdev=83964.41 lat (usec): min=347, max=5857.8K, avg=44185.14, stdev=86640.34 clat percentiles (usec): | 1.00th=[ 1384], 5.00th=[ 3792], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15552], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[168960], 99.90th=[185344], 99.95th=[242688], | 99.99th=[4423680] bw (KB /s): min= 3, max= 1588, per=3.27%, avg=416.27, stdev=74.82 write: io=1023.4MB, bw=413612B/s, iops=100, runt=2594375msec slat (usec): min=6, max=5844.8K, avg=389.10, stdev=37686.19 clat (usec): min=87, max=5855.9K, avg=34582.17, stdev=85968.84 lat (usec): min=97, max=5855.1K, avg=34972.93, stdev=93912.12 clat percentiles (usec): | 1.00th=[ 1560], 5.00th=[ 3888], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[238592], | 99.99th=[4947968] bw (KB /s): min= 1, max= 1395, per=3.27%, avg=415.51, stdev=105.85 lat (usec) : 10=0.01%, 100=0.11%, 250=0.67%, 500=0.02%, 750=0.03% lat (usec) : 1000=0.04% lat (msec) : 2=0.40%, 4=4.13%, 10=22.76%, 20=46.64%, 50=8.70% lat (msec) : 100=0.40%, 250=16.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=267866, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28091: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=410815B/s, iops=100, runt=2615328msec slat (usec): min=209, max=294240, avg=9479.36, stdev=26320.27 clat (usec): min=6, max=6318.3K, avg=34938.98, stdev=87471.40 lat (usec): min=357, max=6323.3K, avg=44420.73, stdev=90146.20 clat percentiles (usec): | 1.00th=[ 1288], 5.00th=[ 3728], 10.00th=[ 6176], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13760], 60.00th=[15680], | 70.00th=[18304], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[187392], 99.95th=[839680], | 99.99th=[4685824] bw (KB /s): min= 3, max= 1584, per=3.27%, avg=415.48, stdev=79.29 write: io=1023.4MB, bw=410298B/s, iops=100, runt=2615328msec slat (usec): min=6, max=6293.5K, avg=472.93, stdev=40611.52 clat (usec): min=87, max=6320.3K, avg=34902.52, stdev=91364.50 lat (usec): min=95, max=6320.3K, avg=35377.05, stdev=99934.18 clat percentiles (usec): | 1.00th=[ 1384], 5.00th=[ 3824], 10.00th=[ 6112], 20.00th=[ 8384], | 30.00th=[10304], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23680], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[185344], 99.95th=[806912], | 99.99th=[4816896] bw (KB /s): min= 3, max= 1384, per=3.27%, avg=415.06, stdev=108.33 lat (usec) : 10=0.01%, 100=0.12%, 250=0.66%, 500=0.03%, 750=0.04% lat (usec) : 1000=0.06% lat (msec) : 2=0.46%, 4=4.16%, 10=22.73%, 20=46.63%, 50=8.64% lat (msec) : 100=0.40%, 250=16.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=0.09%, sys=0.51%, ctx=268200, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28092: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407744B/s, iops=99, runt=2635024msec slat (usec): min=143, max=206589, avg=9394.28, stdev=26244.43 clat (usec): min=6, max=6313.4K, avg=35492.69, stdev=106285.46 lat (usec): min=343, max=6318.7K, avg=44889.36, stdev=108444.63 clat percentiles (usec): | 1.00th=[ 796], 5.00th=[ 3408], 10.00th=[ 5984], 20.00th=[ 8256], | 30.00th=[10176], 40.00th=[11968], 50.00th=[13632], 60.00th=[15552], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[226304], 99.95th=[2473984], | 99.99th=[4816896] bw (KB /s): min= 4, max= 2642, per=3.28%, avg=417.70, stdev=109.36 write: io=1023.4MB, bw=407231B/s, iops=99, runt=2635024msec slat (usec): min=6, max=6301.9K, avg=632.80, stdev=48203.51 clat (usec): min=88, max=6312.8K, avg=34874.57, stdev=96868.73 lat (usec): min=97, max=6314.9K, avg=35509.01, stdev=108188.60 clat percentiles (usec): | 1.00th=[ 780], 5.00th=[ 3504], 10.00th=[ 5920], 20.00th=[ 8096], | 30.00th=[10176], 40.00th=[11840], 50.00th=[13632], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23424], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[191488], 99.95th=[921600], | 99.99th=[4751360] bw (KB /s): min= 1, max= 3047, per=3.28%, avg=417.13, stdev=134.24 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.06%, 750=0.12% lat (usec) : 1000=0.19% lat (msec) : 2=0.98%, 4=4.22%, 10=22.50%, 20=46.42%, 50=8.37% lat (msec) : 100=0.39%, 250=15.88%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.05% cpu : usr=0.09%, sys=0.51%, ctx=268713, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 random-rw: (groupid=0, jobs=1): err= 0: pid=28093: Fri Sep 4 15:05:52 2015 read : io=1024.7MB, bw=407404B/s, iops=99, runt=2637224msec slat (usec): min=123, max=292527, avg=9319.75, stdev=26166.89 clat (usec): min=5, max=5924.4K, avg=35346.10, stdev=105089.94 lat (usec): min=310, max=5927.7K, avg=44668.32, stdev=107296.42 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 3088], 10.00th=[ 5728], 20.00th=[ 8032], | 30.00th=[10048], 40.00th=[11840], 50.00th=[13504], 60.00th=[15424], | 70.00th=[18048], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[166912], 99.50th=[171008], 99.90th=[254976], 99.95th=[2277376], | 99.99th=[4882432] bw (KB /s): min= 3, max= 4228, per=3.30%, avg=419.47, stdev=145.21 write: io=1023.4MB, bw=406892B/s, iops=99, runt=2637224msec slat (usec): min=6, max=5915.1K, avg=716.03, stdev=49145.39 clat (usec): min=88, max=6089.8K, avg=35079.68, stdev=101423.17 lat (usec): min=97, max=6089.9K, avg=35797.30, stdev=112773.01 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 3184], 10.00th=[ 5664], 20.00th=[ 7968], | 30.00th=[10048], 40.00th=[11712], 50.00th=[13504], 60.00th=[15296], | 70.00th=[17792], 80.00th=[23168], 90.00th=[132096], 95.00th=[150528], | 99.00th=[164864], 99.50th=[171008], 99.90th=[228352], 99.95th=[2244608], | 99.99th=[4882432] bw (KB /s): min= 1, max= 4208, per=3.30%, avg=418.84, stdev=168.76 lat (usec) : 10=0.01%, 100=0.12%, 250=0.67%, 500=0.10%, 750=0.24% lat (usec) : 1000=0.34% lat (msec) : 2=1.38%, 4=4.32%, 10=22.46%, 20=45.90%, 50=8.23% lat (msec) : 100=0.41%, 250=15.73%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.06% cpu : usr=0.09%, sys=0.52%, ctx=269232, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=262309/w=261979/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=32789MB, aggrb=12722KB/s, minb=397KB/s, maxb=412KB/s, mint=2543523msec, maxt=2639024msec WRITE: io=32747MB, aggrb=12706KB/s, minb=397KB/s, maxb=411KB/s, mint=2543523msec, maxt=2639024msec Disk stats (read/write): sdb: ios=8393558/7429932, merge=0/868254, ticks=77040645/373166953, in_queue=450199687, util=99.83%