Details
Description
If running a repairDatabase we sometimes see an invariant failure where looking for file:collection... returns WT_NOTFOUND (which gets translated to EINVAL).
Issue Links
- is related to
-
WT-3024
wtperf medium-lsm-compact test can hang
-
- Resolved
-
- links to
Activity
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2984 Keep sufficient history in the metadata for queries. (#3144)
WT-2984Keep sufficient history in the metadata for queries.
Since we treat the checkpoint transaction specially, we also have to
track the amount of history required for the metadata specially.
Previously, there was a window where a query started while a checkpoint
was running could fail to see the results of the checkpoint when it
queried the metadata.
- Remove the unused "session" argument to WT_IS_METADATA.
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/adfd8056806c640a8d8aaba8766d07dd5fc79569
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: Import wiredtiger: ca6eee06ffdacc8e191987e64b3791740dad21e1 from branch mongodb-3.4
ref: 74430da40c..ca6eee06ff
for: 3.4.0
WT-2962 Provide a way to configure builtin extensions
WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND
WT-3000 Missing log records in recovery when crashing after a log file switch
WT-3002 Allow applications to exempt threads from eviction.
WT-3004 lint: declare functions that don't return a value as void
WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor.
WT-3012 Test format hanging on LSM configurations
WT-3015 Test format stuck with 2mb cache
WT-3016 Tests needed for systems without ftruncate
WT-3017 Hazard pointer race with page replace causes error
WT-3018 lint
WT-3020 LSM primary changes impact parallel-pop-lsm load time
WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete
WT-3023 Test format hang on zSeries
WT-3024 wtperf medium-lsm-compact test can hang
Branch: master
https://github.com/mongodb/mongo/commit/fb4ae3792065e98696e391ac1c4602216b8502cb
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2984 Keep sufficient history in the metadata for queries. (#3144)
WT-2984Keep sufficient history in the metadata for queries.
Since we treat the checkpoint transaction specially, we also have to
track the amount of history required for the metadata specially.
Previously, there was a window where a query started while a checkpoint
was running could fail to see the results of the checkpoint when it
queried the metadata.
- Remove the unused "session" argument to WT_IS_METADATA.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/adfd8056806c640a8d8aaba8766d07dd5fc79569
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2984 Keep sufficient history in the metadata for queries. (#3144)
WT-2984Keep sufficient history in the metadata for queries.
Since we treat the checkpoint transaction specially, we also have to
track the amount of history required for the metadata specially.
Previously, there was a window where a query started while a checkpoint
was running could fail to see the results of the checkpoint when it
queried the metadata.
- Remove the unused "session" argument to WT_IS_METADATA.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/adfd8056806c640a8d8aaba8766d07dd5fc79569
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: Import wiredtiger: 040e3d6f764c0fb626cb47fede54469f57d0c6e0 from branch mongodb-3.2
ref: 187707a5c1..040e3d6f76
for: 3.2.12
WT-2962 Provide a way to configure builtin extensions
WT-2984 Search of metadata for recently created collection gets WT_NOTFOUND
WT-3000 Missing log records in recovery when crashing after a log file switch
WT-3002 Allow applications to exempt threads from eviction.
WT-3004 lint: declare functions that don't return a value as void
WT-3011 __wt_curjoin_open() saves the wrong URI in the cursor.
WT-3012 Test format hanging on LSM configurations
WT-3015 Test format stuck with 2mb cache
WT-3016 Tests needed for systems without ftruncate
WT-3017 Hazard pointer race with page replace causes error
WT-3018 lint
WT-3020 LSM primary changes impact parallel-pop-lsm load time
WT-3022 LSM operations get stuck in __wt_clsm_await_switch waiting for switch on tree to complete
WT-3023 Test format hang on zSeries
WT-3024 wtperf medium-lsm-compact test can hang
Branch: v3.2
https://github.com/mongodb/mongo/commit/c586934f7212f6a9a2087cbaf9a8fcd7d7ce9abf
I've been looking into this. I can reproduce within 10-15 minutes on my AWS box. The command I'm using to reproduce is:
LD_LIBRARY_PATH=$WT_INSTALL/lib:$LD_LIBRARY_PATH python27 buildscripts/resmoke.py --repeat=5000 jstests/core/repair_database.js jstests/core/profile_insert.jsI added an abort when we get the WT_NOTFOUND in schema_source_config. Here's the stack:
#1 0x00007fdc22a5dce8 in abort () from /lib64/libc.so.6#2 0x00007fdc239e0b79 in __schema_source_config (session=0x561b6c13a400, srch=0x561b79bca800,config=0x561b796d28d4 "app_metadata=(formatVersion=1),collator=,columns=,source=\"file:collection-70590-5052041337794129251.wt\",type=file",result=0x7fdc1a1f19e8) at ../src/cursor/cur_metadata.c:47#3 0x00007fdc239e0de9 in __schema_create_collapse (session=0x561b6c13a400, mdc=0x561b797b4780, key=0x561b701351ae "collection-70590-5052041337794129251",value=0x561b798a7714 "app_metadata=(formatVersion=1),colgroups=,collator=,columns=,key_format=q,value_format=u", value_ret=0x7fdc1a1f1a78)at ../src/cursor/cur_metadata.c:89#4 0x00007fdc239e1098 in __schema_create_collapse (session=0x2b, mdc=0x561b76282690, key=0x561b7535e000 "", value=0x0, value_ret=0x561b798a7714)at ../src/cursor/cur_metadata.c:116#5 0x00007fdc239e28d8 in __curmetadata_search (cursor=0x561b797b4780) at ../src/cursor/cur_metadata.c:366f#6 0x0000561b697406c6 in mongo::WiredTigerUtil::getMetadata[abi:cxx11](mongo::OperationContext*, mongo::StringData) (opCtx=opCtx@entry=0x561b7265e700, uri=...)at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp:111#7 0x0000561b697449d5 in mongo::WiredTigerUtil::getApplicationMetadata (opCtx=opCtx@entry=0x561b7265e700, uri=..., bob=bob@entry=0x7fdc1a1f1ed0)at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp:130I added some messages to WT to see what the LSN is when the create returns to the user and what the LSN is just before we call srch->search in schema_source_config. Here is the output:
[MongoDFixture:job0] 2016-10-18T19:24:25.763+0000 E STORAGE [conn7434] WiredTiger error (0) [1476818665:763370][1238:0x7fdc1a1f5700], WT_SESSION.create: CREATE: Created table:collection-70590-5052041337794129251. LSN [3][14341248]...[MongoDFixture:job0] 2016-10-18T19:24:25.782+0000 I STORAGE [conn7434] Repairing collection repairDB.a[MongoDFixture:job0] 2016-10-18T19:24:26.107+0000 I STORAGE [conn7434] Verify succeeded on uri table:collection-70590-5052041337794129251. Not salvaging.[MongoDFixture:job0] 2016-10-18T19:24:26.107+0000 E STORAGE [conn7434] WiredTiger error (0) [1476818666:107293][1238:0x7fdc1a1f5700], file:WiredTiger.wt, WT_CURSOR.search: SCHEMA: Cannot find file:collection-70590-5052041337794129251.wt. LSN [3][14357248][MongoDFixture:job0] 2016-10-18T19:24:26.118+0000 F - [conn7434] Got signal: 6 (Aborted).All three entries (table, file, colgroup) into the metadata for collection-70590 and a file sync of the metadata file were done before LSN 3,14357248. But there were no entries yet entered when create returned at 3,14341248.