Problem in the Named Pipe feature PM-2971: when an aggregation using $_externalDataSources is done, one or more virtual collections is temporarily created in the memory portion of the catalog only, then implicitly dropped when the query is complete. However, the implicit drops are happening asynchronously, so if another $_externalDataSources query is done immediately after using the same virtual collection name(s), it may nondeterministically fail with an error like

uncaught exception: Error: command failed: {
	"ok" : 0,
	"errmsg" : "Collection test.coll already exists.",
	"code" : 48,
	"codeName" : "NamespaceExists"
}

The following simple test in the mongo shell will reproduce it. (If 10 loop iterations is not enough, just increase it. The failure is usually hit in the first 10 iterations for me.)

const kUrlProtocolFile = "file://";
const kPipeName = "pipeName";
for (let i = 0; i < 10; ++i) {
    _writeTestPipeBsonFile(kPipeName, 1000, "hits.10.bson");
    db.coll.aggregate([], {
        $_externalDataSources: [{
            collName: "coll",
            dataSources: [
                {url: kUrlProtocolFile + kPipeName, storageType: "pipe", fileType: "bson"},
            ]
        }]
    });
}

If you add an explicit

db.coll.drop();

to the end of the loop, the failures are averted, but then the mongod console fills up with a mixture of messages about deferring the drop, completing the drop, and failing the drop, the last of which looks like

"s":"E",  "c":"QUERY",    "id":6968700, "ctx":"conn11","msg":"Failed to drop an external data source","attr":{"coll":"test.coll"}}

The above repro uses the data set hits.10.bson which can be found here:

https://drive.google.com/drive/folders/1xC_aAtY_W8Hn5zQq5n7opd5N4NBz1lmq

FYI arun.banala@mongodb.com