-
Type:
Build Failure
-
Resolution: Gone away
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
-
8
A test timed out after over two hours, the call stacks just prior to shutdown indicate that threads are waiting on pages to be read, while a thread is doing a forced eviction:
[2020/05/28 09:45:55.077] 00 (Inline Function) --------`-------- mongod!__wt_update_list_memsize+0x17 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\include\btree.i @ 494] [2020/05/28 09:45:55.077] 01 0000008f`a2ef87c0 00007ff7`2e268910 mongod!__wt_row_modify(struct __wt_cursor_btree * cbt = 0x0000008f`a2ef8900, struct __wt_item * key = 0x000001c2`f3f18060, struct __wt_item * value = 0x00000000`00000000, struct __wt_update * upd_arg = 0x000001c2`a7ef5270, unsigned int modify_type = 0, bool exclusive = true)+0x2e3 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\row_modify.c @ 114] [2020/05/28 09:45:57.585] 02 0000008f`a2ef88a0 00007ff7`2e26a8eb mongod!__split_multi_inmem(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_page * orig = 0x000001c2`85a79000, struct __wt_multi * multi = 0x000001c2`a7836d80, struct __wt_ref * ref = 0x000001c2`8615d080)+0x2d0 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_split.c @ 1502] [2020/05/28 09:45:57.585] 03 0000008f`a2ef8d20 00007ff7`2e22e38b mongod!__wt_split_rewrite(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_ref * ref = 0x000001c2`a5739f80, struct __wt_multi * multi = 0x000001c2`a7836d80)+0x7b [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_split.c @ 2260] [2020/05/28 09:45:57.585] 04 0000008f`a2ef8d60 00007ff7`2e22ed5f mongod!__evict_page_dirty_update(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_ref * ref = 0x000001c2`a5739f80, unsigned int evict_flags = 4)+0x28b [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\evict\evict_page.c @ 409] [2020/05/28 09:46:00.862] 05 0000008f`a2ef8e50 00007ff7`2e22f0c5 mongod!__wt_evict(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_ref * ref = 0x000001c2`a5739f80, unsigned char previous_state = 0x03 '', unsigned int flags = <Value unavailable error>)+0x5ef [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\evict\evict_page.c @ 230] [2020/05/28 09:46:00.862] 06 0000008f`a2ef8ef0 00007ff7`2e23b891 mongod!__wt_page_release_evict(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_ref * ref = 0x000001c2`a5739f80, unsigned int flags = <Value unavailable error>)+0xa5 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\evict\evict_page.c @ 82] [2020/05/28 09:46:00.862] 07 0000008f`a2ef8f30 00007ff7`2e230f54 mongod!__wt_page_in_func(struct __wt_session_impl * session = 0x000001c2`dfa32dc0, struct __wt_ref * ref = 0x000001c2`a5739f80, unsigned int flags = <Value unavailable error>)+0x301 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_read.c @ 348] [2020/05/28 09:46:00.862] 08 (Inline Function) --------`-------- mongod!__wt_page_swap_func+0x2d [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\include\btree.i @ 1649] [2020/05/28 09:46:03.844] 09 0000008f`a2ef8ff0 00007ff7`2e228854 mongod!__wt_row_search(struct __wt_cursor_btree * cbt = 0x000001c2`eb9aac00, struct __wt_item * srch_key = 0x000001c2`eb9aad00, bool insert = <Value unavailable error>, struct __wt_ref * leaf = 0x000001c2`85308798, bool leaf_safe = false, bool * leaf_foundp = 0x00000000`00000000)+0x5f4 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\row_srch.c @ 423] [2020/05/28 09:46:03.844] 0a 0000008f`a2ef90d0 00007ff7`2e22b89a mongod!__cursor_row_search(struct __wt_cursor_btree * cbt = 0x000001c2`eb9aac00, bool insert = false, struct __wt_ref * leaf = 0x00000000`00000000, bool * leaf_foundp = 0x00000000`00000000)+0x64 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_cursor.c @ 367] [2020/05/28 09:46:03.844] 0b 0000008f`a2ef9120 00007ff7`2e22a2a4 mongod!__wt_btcur_search(struct __wt_cursor_btree * cbt = 0x000001c2`eb9aac00)+0x42a [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_cursor.c @ 547] [2020/05/28 09:46:06.895] 0c 0000008f`a2ef91b0 00007ff7`2e1c92cd mongod!__wt_btcur_modify(struct __wt_cursor_btree * cbt = 0x000001c2`eb9aac00, struct __wt_modify * entries = 0x000001c2`ed2b7e70, int nentries = 0n1)+0xd4 [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\btree\bt_cursor.c @ 1443] [2020/05/28 09:46:06.895] 0d 0000008f`a2ef9240 00007ff7`2cf2d0f0 mongod!__curfile_modify(struct __wt_cursor * cursor = 0x000001c2`eb9aac00, struct __wt_modify * entries = 0x000001c2`ed2b7e70, int nentries = 0n1)+0x59d [C:\data\mci\15fbaacd55f18669e5ab479ace02f8a9\src\src\third_party\wiredtiger\src\cursor\cur_file.c @ 333]
We should see if we can figure out whether the forced evictions are continually failing, or whether this single eviction has been running for an unreasonably large amount of time. Also worth checking whether the test normally runs close to the timeout - there might not be any problem here at all.
The test is running an inMemory workload - it's possible that the test is requiring more history than fits in cache, which means that we are constantly trying to consolidate the amount of space being used without making progress.