Michael, I saw a free-page race today.
Here's the gdb run:
1: read/write ops 12570 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7a04700 (LWP 1961)] 0x000000000041a5df in __hazard_exclusive (session=0x85ce20, ref=0x0) at ../src/btree/rec_evict.c:500 500 if (WT_ATOMIC_CAS(ref->state, WT_REF_EVICTING, WT_REF_LOCKED)) Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86 _64 glibc-2.12-1.47.el6_2.5.x86_64 (gdb) p ref $1 = (WT_REF *) 0x0 (gdb) where #0 0x000000000041a5df in __hazard_exclusive (session=0x85ce20, ref=0x0) at ../src/btree/rec_evict.c:500 WT-1 0x000000000041a3aa in __rec_review (session=0x85ce20, page=0x994940, flags=0, top=1) at ../src/btree/rec_evict.c:359 WT-2 0x0000000000419c18 in __wt_rec_evict (session=0x85ce20, page=0x994940, flags=0) at ../src/btree/rec_evict.c:47 WT-3 0x0000000000410c72 in __evict_request_walk (session=0x85ce20) at ../src/btree/bt_evict.c:367 WT-4 0x0000000000410a08 in __evict_worker (session=0x85ce20) at ../src/btree/bt_evict.c:284 WT-5 0x00000000004108cd in __wt_cache_evict_server (arg=0x88f5e0) at ../src/btree/bt_evict.c:242 WT-6 0x00000030f38077f1 in start_thread () from /lib64/libpthread.so.0 WT-7 0x00000030f34e592d in clone () from /lib64/libc.so.6 (gdb) info threads * 4 Thread 0x7ffff7a04700 (LWP 1961) 0x000000000041a5df in __hazard_exclusive (session=0x85ce20, ref=0x0) at ../src/btree/rec_evict.c:500 1 Thread 0x7ffff7fed700 (LWP 1946) 0x00000031048098fb in BZ2_decompress () from /lib64/libbz2.so.1
It's not real reliable, but I found a run that reproduces it most of the time.
evictserver: sleeping
evictserver: waking
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {0x7fca541173d0}
evictserver: sleeping
evictserver: waking
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {0x7fca541173d0}
evictserver: sleeping
evictserver: waking
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {0x7fca541173d0}
evictserver: sleeping
evictserver: waking
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {0x7fca541173d0}
evictserver: sleeping
evictserver: waking
file:__wt [cursor.insert]: evictserver: ../src/include/cache.i/59: page 0xf3d8a0
{0x7fca541173d0}
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {0x7fca541173d0}
file:__wt [cursor.insert]: evictserver: waking, bytes inuse <= max (0MB <= 30MB)
,
file:__wt: evictserver: forcing eviction of page 0xf3d8a0 {(nil)}
Somehow we're evicting a page, discarding it, and then evicting it again, and we go south because WT_PAGE->ref is NULL.
I changed __wt_page_out() to overwrite the page on discard with 0xab, and the core shows me something more interesting:
(gdb) p page
$1 = (WT_PAGE *) 0x9359e0
(gdb) p *page
$2 = {parent = 0x983750, ref = 0xabababababababab, u = {intl = {recno = 12370169
555311111083, t = 0xabababababababab},
row = {d = 0xabababababababab, ins = 0xabababababababab, upd = 0xabababababa
babab}, col_fix = {
recno = 12370169555311111083, bitf = 0xabababababababab <Address 0xabababa
babababab out of bounds>}, col_var = {
recno = 12370169555311111083, d = 0xabababababababab, repeats = 0xabababab
abababab, nrepeats = 2880154539}},
dsk = 0xabababababababab, modify = 0xabababababababab, read_gen = 123701695553
11111083, entries = 2880154539,
memory_footprint = 2880154539, type = 171 '\253', flags = 171 '\253'}
(gdb)
I'll push that change, we might as well keep it.
I put a check in __wt_evict_page_request:
WT_EVICT_REQ_FOREACH(er, er_end, cache)
if (er->session != NULL && er->page == page)
fprintf(stderr, "WT-1: %p {%d} %p {%d}\n",
er->page, er->page->ref->state,
page, page->ref->state);
if (!WT_ATOMIC_CAS(page->ref->state, WT_REF_MEM, WT_REF_EVICTING))
return (0);
WT_EVICT_REQ_FOREACH(er, er_end, cache)
if (er->session != NULL && er->page == page)
fprintf(stderr, "WT-2 %p {%d} %p {%d}\n",
er->page, er->page->ref->state,
page, page->ref->state);
which gives me:
./t -r
... reading CONFIG file
t: process 3060
WT-1: 0x11921e0 {3} 0x11921e0 {3}
WT-2 0x11921e0 {1} 0x11921e0 {1}
Segmentation fault
which translates to both page and er->page being set to WT_REF_MEM, then both set to WT_REF_EVICTING.
I haven't gotten any further, but if I don't get back to it today, maybe it will be obvious to you?
The run file is:
############################################ # RUN PARAMETERS ############################################ # bitcnt not applicable to this run bzip=1 cache=30 delete_pct=29 file_type=row-store huffman_key=0 huffman_value=1 insert_pct=9 internal_page_max=11 key_max=121 key_min=17 leaf_page_max=9 ops=20000 # repeat_data_pct not applicable to this run reverse=1 rows=100 runs=1 value_max=1805 value_min=17 write_pct=70 ############################################
- is related to
-
WT-104 memory barrier after reading page's write generation
- Closed
-
- Closed
- related to
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed
-
- Closed