[SERVER-53594] Allow collections and databases to be dropped in standalone mode with unfinished replicated index builds Created: 05/Jan/21  Updated: 29/Oct/23  Resolved: 17/Oct/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.3
Fix Version/s: 6.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: Vitalii K. Assignee: Shin Yee Tan
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

Steps to Reproduce

MongoDB shell version v4.4.3

> db.zz_10521_1605225225_1.drop()
uncaught exception: Error: error doing query: failed: network error while attempting to run command 'drop' on host '10.10.10.13:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:169:19
DBCollection.prototype.drop@src/mongo/shell/collection.js:709:11
@(shell):1:1

/var/log/mongodb/mongod.log

{"t":{"$date":"2021-01-05T22:08:24.599+02:00"},"s":"F",  "c":"-",        "id":23079,   "ctx":"conn2","msg":"Invariant failure","attr":{"expr":"coll->getIndexCatalog()->numIndexesInProgress(opCtx) == 0","file":"src/mongo/db/catalog/drop_collection.cpp","line":214}}
{"t":{"$date":"2021-01-05T22:08:24.599+02:00"},"s":"F",  "c":"-",        "id":23080,   "ctx":"conn2","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-01-05T22:08:24.599+02:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn2","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

Expected Results

deleted collection

Actual Results

uncaught exception: Error: error doing query: failed: network error while attempting to run command 'drop' on host '10.10.10.13:27017' :

Sprint: Execution Team 2022-10-17, Execution Team 2022-10-31
Participants:

 Description   

MongoDB Standalone Server with authorization. Proxmox, Ubuntu LXC container.

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

db version v4.4.3
Build Info: {
"version": "4.4.3",
"gitVersion": "913d6b62acfbb344dde1b116f4161360acd8fd13",
"openSSLVersion": "OpenSSL 1.1.1 11 Sep 2018",
"modules": [],
"allocator": "tcmalloc",
"environment":

{ "distmod": "ubuntu1804", "distarch": "x86_64", "target_arch": "x86_64" }

}

Problem Description

there were 20 collections in the current database; 3 are not deleted, error; after an attempt to delete, there is no way to connect to the database; only restart helps



 Comments   
Comment by Githook User [ 14/Oct/22 ]

Author:

{'name': 'Shin Yee Tan', 'email': 'shinyee.tan@mongodb.com', 'username': 'shinyeet'}

Message: SERVER-53594 Add frozen indexes to total index counts and subtract from in progress index counts
Branch: master
https://github.com/mongodb/mongo/commit/8c0cd9eb1e2e32b489b5d93ba9ccb18b0aa43cf0

Comment by Eric Sedor [ 12/Jan/21 ]

Understood vkatochka@protonmail.com, we'll keep this ticket open for now in case this comes up again soon. Thank you!

Comment by Vitalii K. [ 12/Jan/21 ]

The Replica Set was modified two months ago. Now there is no way to do this on this server. I'll try to replicate this from scratch on another server, can't promise when.

Comment by Eric Sedor [ 06/Jan/21 ]

Thanks vkatochka@protonmail.com, that seems to reinforce Louis's suspicion that an index build may have been interrupted by transitioning from a replica set to standalone mode. We're interested in addressing this as a bug.

Can you let us know if you can:

1) restart the node as a replica set
2) wait for any index builds to complete
3) return to standalone mode
4) drop the database

If this doesn't work can you provide a complete log file covering these activities, as well as archive (tar or zip) of the $dbpath/diagnostic.data directory (the contents are described here)?

Thanks!

Comment by Vitalii K. [ 06/Jan/21 ]

Mmm ... I have a standalone server running with no replica set.  But when starting mongo shell (when there was an error) I was getting the following message:
~~~
2021-01-05T22:27:07.985+02:00: Document(s) exist in 'system.replset', but started without --replSet. Database contents may appear inconsistent with the writes that were visible when this node was running as part of a replica set. Restart with --replSet unless you are doing maintenance and no other clients are connected. The TTL collection monitor will not start because of this. For more info see http://dochub.mongodb.org/core/ttlcollections
~~~

Previously, some work was done related to oplog and possibly some options related to replSet were included. And it is possible that collections and indexes were created during this period.

Comment by Louis Williams [ 05/Jan/21 ]

Hi vkatochka@protonmail.com, thanks for filing this bug report. Did you by any chance start an index build while the node was in a replica set and shut it down before the index build finished? If so, you should also be able to drop the database (without crashing) if you restart the node as a replica set member again (with --replSet).

Comment by Vitalii K. [ 05/Jan/21 ]

After dropping the index from the collection, the collection was dropped successfully.

Comment by Vitalii K. [ 05/Jan/21 ]

tried to delete db

> db.dropDatabase()
uncaught exception: Error: error doing query: failed: network error while attempting to run command 'dropDatabase' on host '10.10.10.13:27017' :
DB.prototype.runCommand@src/mongo/shell/db.js:169:19
DB.prototype.dropDatabase@src/mongo/shell/db.js:400:12

 

{"t":{"$date":"2021-01-05T22:31:14.321+02:00"},"s":"F", "c":"-", "id":23081, "ctx":"conn31","msg":"Invariant failure","attr":{"expr":"!coll->getIndexCatalog()->haveAnyIndexesInProgress()","msg":"An index is building on collection 'mn.zz_10521_1605225225_1'.","file":"src/mongo/db/catalog/database_holder_impl.cpp","line":191}}
{"t":{"$date":"2021-01-05T22:31:14.321+02:00"},"s":"F", "c":"-", "id":23082, "ctx":"conn31","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-01-05T22:31:14.321+02:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn31","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

 

 

Generated at Thu Feb 08 05:31:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.