Details
Bug
Major - P3
Description
We need a mechanism to allow a backup cursor to create a backup file name list without actually opening every file in the list. Related to SERVER-26595 but needs both a MongoDB change and a WT change.
Issue Links
Activity
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2968 Fix a segfault when a drop races with closing a backup cursor. (#3127)
We were clearing the shared pointer to the list of filenames in the
backup, then releasing the hotbackup lock before clearing the hotbackup
flag. That means drops need to check both the flag and that the pointer
is non-NULL.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/2fe91f73f7def64508c945a7dfaac03fd9e8495c
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2968 Fix a segfault when a drop races with closing a backup cursor. (#3127)
We were clearing the shared pointer to the list of filenames in the
backup, then releasing the hotbackup lock before clearing the hotbackup
flag. That means drops need to check both the flag and that the pointer
is non-NULL.
Branch: mongodb-3.4
https://github.com/wiredtiger/wiredtiger/commit/2fe91f73f7def64508c945a7dfaac03fd9e8495c
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: Import wiredtiger: 74430da40c96247c0a0d92f011ccfb2b175f1f32 from branch mongodb-3.4
ref: ef9a7983ea..74430da40c
for: 3.4.0
SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
WT-2415 Add option for join cursors to return false positives
WT-2945 Occasional hang running reconfigure fuzz test
WT-2947 test suite populate functions need refactoring
WT-2953 Test format failure - stuck cache with 1/42MB dirty
WT-2961 Add a version drop-down to the web version of the docs
WT-2964 WTPERF parallel-pop-btree hung in populate phase
WT-2968 backup cursor opens every file
WT-2971 Add documentation for raw compression to the WT docs
WT-2975 Reduce the memory allocated for statistics
WT-2991 Coverity issue 1365198
WT-2998 Add error messages to error returns that might be confusing.
WT-2999 Memory leak in join WT_CURSOR with custom extractor
WT-3001 WT_EXTENSION_API references are named inconsistently.
WT-3003 log generation should not auto generate the record and op types
WT-3005 Add top-level .gitignore file.
WT-3007 Remove duplicated word in comment
WT-3008 Run wtperf stress testing against all maintained branches
WT-3009 Test format hung/aborted with small cache (sub 10mB)
Branch: master
https://github.com/mongodb/mongo/commit/0f0db45770f45eab89bd7c2530a259f5ed5b6334
Author:
{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: WT-2968 Fix a segfault when a drop races with closing a backup cursor. (#3127)
We were clearing the shared pointer to the list of filenames in the
backup, then releasing the hotbackup lock before clearing the hotbackup
flag. That means drops need to check both the flag and that the pointer
is non-NULL.
Branch: mongodb-3.2
https://github.com/wiredtiger/wiredtiger/commit/2fe91f73f7def64508c945a7dfaac03fd9e8495c
Author:
{u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}Message: Import wiredtiger: 2e2d5fe23badac038944bdc6639eb62d77136f1a from branch mongodb-3.2
ref: b11ed312ce..2e2d5fe23b
for: 3.2.11
SERVER-26753 Minor speed regression (13%) and 'choppy' performance in 3.4 vs 3.2
WT-2415 Add option for join cursors to return false positives from a Bloom filter
WT-2945 Occasional hang running reconfigure fuzz test
WT-2947 test suite populate functions need refactoring
WT-2953 Test format failure - stuck cache with 1/42MB dirty
WT-2961 Add a version drop-down to the web version of the docs
WT-2964 WTPERF parallel-pop-btree hung in populate phase
WT-2968 backup cursor opens every file
WT-2971 Add documentation for raw-compression to the tune_compression page in WT docs
WT-2975 Reduce the memory allocated for statistics
WT-2991 Coverity issue 1365198
WT-2998 Add error messages to error returns that might be confusing.
WT-2999 Memory leak in join WT_CURSOR with custom extractor
WT-3001 WT_EXTENSION_API references are named inconsistently.
WT-3003 log generation should not auto generate the record and op types
WT-3005 Add top-level .gitignore file.
WT-3007 Remove duplicated word in comment
WT-3008 Run wtperf stress testing against all maintained branches
WT-3009 Test format hung/aborted with small cache (sub 10mB)
Branch: v3.2
https://github.com/mongodb/mongo/commit/7c626977cf052314d09648d1061e07c44e964264
Tracking this down further. My first suggested solution of just sending in a well-known target does not work because cur_backup.c will only add that target and the WT files to the WiredTiger.backup file. So when the user doing fsyncLock; cp/rsync; fsyncUnlock runs on the backup directory, the metadata created from the backup file will not be correct. We need a mechanism to tell WT to create the correct backup file, but not to open all the files.
The file open is done in backup_list_append. The reason for this is to prevent schema operations from removing the file, for example. With fsyncLock MongoDB forbids all modifications already. I overloaded the readonly config setting and added use of that setting in MongoDB to add the name to the backup file but not to open the physical file.