[SERVER-23131] validate() can return EBUSY on WiredTiger Created: 14/Mar/16  Updated: 06/Dec/22  Resolved: 10/Jun/19

Status: Closed
Project: Core Server
Component/s: Diagnostics, WiredTiger
Affects Version/s: 3.2.4, 3.3.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Backlog - Storage Execution Team
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-27345 reclaim inactive WT sessions on calli... Closed
Assigned Teams:
Storage Execution
Operating System: ALL
Steps To Reproduce:

var replTest = new ReplSetTest({nodes: 1});
var version = "latest";
var storageEngine = "wiredTiger";
var nodes = replTest.startSet({binVersion: version, storageEngine: storageEngine});
replTest.initiate();
var db = replTest.getPrimary().getDB("test");
print(tojson(db.serverStatus()));
var res = db.adminCommand('listDatabases');
var invalidColls = [];
var ebusyColls = [];
var totalNumEbusy = 0;
var ebusyCollectionWait = 2000;
var ebusyCollectionLoops = 30;
var ebusyAllLoops = 300000;
res.databases.forEach(dbInfo => {
   var colls = db.getSiblingDB(dbInfo.name).getCollectionInfos();
   colls.forEach(coll => {
      print(dbInfo.name, coll.name);
      var numEbusy = 0;
      var timeStart = new Date().getTime();
      while (true) {
         var val = db.getSiblingDB(dbInfo.name)[coll.name].validate(true);
         var ebusy = val.errors.toString().match('EBUSY');
         if (!ebusy ||
             ++totalNumEbusy >= ebusyAllLoops ||
             ++numEbusy >= ebusyCollectionLoops) {
            if (ebusy) {
               ebusyColls.push(val.ns);
            }
            break;
         }
         sleep(ebusyCollectionWait);
      }
      var timeEnd = new Date().getTime();
      print(tojson(val), '\ntime to validate(msecs)', timeEnd - timeStart);
      if (!val.valid) {
         invalidColls.push(val.ns);
      }
   });
});
if (ebusyColls.length) {
   print('EBUSY collections', ebusyColls.join(' '));
}
if (invalidColls.length) {
   print('Invalid collections', invalidColls.join(' '));
}
replTest.stopSet();

Sprint: TIG 12 (04/01/16)
Participants:

 Description   

When performing a collection validate on all databases/collections (for wiredTiger), some collections will return an EBUSY warning.

Note - this does not happen in 3.0.10



 Comments   
Comment by Eric Milkie [ 10/Jun/19 ]

With the changes in cursor caching, we believe this warning should no longer happen (except on the oplog which uses special locking).

Comment by Cliff Kelley [ 08/Dec/16 ]

Thanks Robert -

Appreciate the clarification. Was assuming error.
Enjoy your holidays!
Cliff

Comment by Robert Guo (Inactive) [ 08/Dec/16 ]

Hi efx_account, thank you for your inquiry. Unfortunately there's no workaround at the moment.

Note that the original ticket description of "Collection Validate can fail with EBUSY" is incorrect. The "verify() returned EBUSY" message is a warning and should not cause the validate() command to fail. I've updated the ticket description to be more accurate.

If your validate() command is failing and/or returning {ok: 0}, it is likely there's a legitimate issue your data.

Comment by Cliff Kelley [ 29/Nov/16 ]

Seeing the same issue in 3.2.9
"errors" : [
"verify() returned EBUSY. Not treating as invalid."

any updates on this?
workaround?

Generated at Thu Feb 08 04:02:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.