[SERVER-71809] $_externalDataSources implicit virtual collection drop is asynchronous Created: 02/Dec/22  Updated: 27/Oct/23  Resolved: 05/Dec/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kevin Cherkauer Assignee: Kevin Cherkauer
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Operating System: ALL
Sprint: QE 2022-12-12
Participants:

 Description   

Problem in the Named Pipe feature PM-2971: when an aggregation using $_externalDataSources is done, one or more virtual collections is temporarily created in the memory portion of the catalog only, then implicitly dropped when the query is complete. However, the implicit drops are happening asynchronously, so if another $_externalDataSources query is done immediately after using the same virtual collection name(s), it may nondeterministically fail with an error like

uncaught exception: Error: command failed: {
	"ok" : 0,
	"errmsg" : "Collection test.coll already exists.",
	"code" : 48,
	"codeName" : "NamespaceExists"
}

The following simple test in the mongo shell will reproduce it. (If 10 loop iterations is not enough, just increase it. The failure is usually hit in the first 10 iterations for me.)

const kUrlProtocolFile = "file://";
const kPipeName = "pipeName";
for (let i = 0; i < 10; ++i) {
    _writeTestPipeBsonFile(kPipeName, 1000, "hits.10.bson");
    db.coll.aggregate([], {
        $_externalDataSources: [{
            collName: "coll",
            dataSources: [
                {url: kUrlProtocolFile + kPipeName, storageType: "pipe", fileType: "bson"},
            ]
        }]
    });
}

If you add an explicit

db.coll.drop();

to the end of the loop, the failures are averted, but then the mongod console fills up with a mixture of messages about deferring the drop, completing the drop, and failing the drop, the last of which looks like

"s":"E",  "c":"QUERY",    "id":6968700, "ctx":"conn11","msg":"Failed to drop an external data source","attr":{"coll":"test.coll"}}

The above repro uses the data set hits.10.bson which can be found here:

https://drive.google.com/drive/folders/1xC_aAtY_W8Hn5zQq5n7opd5N4NBz1lmq

FYI arun.banala@mongodb.com 



 Comments   
Comment by Kevin Cherkauer [ 05/Dec/22 ]

Adding .toArray() makes the problem go away.

Thus I think there is no bug here after all – it is just the normal behavior of doing an aggregation and not consuming the cursor.

Thank you, arun.banala@mongodb.com ! Closing this.

Generated at Thu Feb 08 06:20:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.