[SERVER-5264] Database crached (out of space while loading data). Killed processes, and retsrated mongo. Still see my database, but can not get rid of it. Created: 09/Mar/12  Updated: 15/Aug/12  Resolved: 11/Mar/12

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: George Nikopoulos Assignee: Gregor Macadam
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 64 bit


Attachments: Text File from_config.txt     Text File from_mongos.txt     Text File from_shard1.txt     Text File from_shard2.txt    
Participants:

 Description   

database crashed (out of disk space) yesterday, then I restarted it. I see nothing loged in the log
file since the crash.

See my actions below:
Set up sharding:oracle@dw07orl57 > mongo
MongoDB shell version: 2.0.3
connecting to: test
mongos> use admin
switched to db admin
mongos> show dbs
config 0.1875GB
gis_db 1.953125GB
mongos> use gis_db
switched to db gis_db
mongos> db.dropdatabase()
Fri Mar 9 10:29:57 TypeError: db.dropdatabase is not a function (shell):1
mongos> db.dropDatabase()

{ "assertion" : "collection's metadata is undergoing changes. Please try again.", "assertionCode" : 13331, "errmsg" : "db assertion failure", "ok" : 0 } mongos> db.dropDatabase() { "dropped" : "gis_db", "ok" : 1 }

mongos> show dbs
config 0.1875GB
gis_db 1.953125GB



 Comments   
Comment by George Nikopoulos [ 19/Mar/12 ]

Even though within the database I would do, "show dbs", and then for gis_db database I do a db.dropDatabase() successfully, next time I run "show dbs" gis_db would still show up (empty). Once I deleted the OS gis_db related files and bounce the database, it cleaned up. Now when I do show dbs, the gis_db does not show up.

Comment by George Nikopoulos [ 12/Mar/12 ]

Thanks for you responce Eliot. I tried again today and it seems to me that I have the same problem. This is a test database, so I could rebuild/reinstall it, I am just worrying (what if it was production). The behavior is weird: The first time i get the "metadata is undergoing changes" message , then it allows me to drop the database, and then the dataabse still show up. I will attach all three log files from mongos, and shard1, and shard2. The phycical database files are also still on the disk as: oracle@dw07orl57 > pwd
/app/data/db/shard2
oracle@dw07orl57 > ls -ltr
total 2050040
drwxr-xr-x 3 oracle oinstall 4096 Mar 8 15:45 moveChunk
rw------ 1 oracle oinstall 16777216 Mar 8 16:35 gis_db.ns
rw------ 1 oracle oinstall 67108864 Mar 8 16:35 gis_db.0
rw------ 1 oracle oinstall 134217728 Mar 8 16:35 gis_db.1
rw------ 1 oracle oinstall 268435456 Mar 8 16:45 gis_db.2
rw------ 1 oracle oinstall 536870912 Mar 8 16:45 gis_db.3
rw------ 1 oracle oinstall 1073741824 Mar 8 16:47 gis_db.4
-rwxr-xr-x 1 oracle oinstall 6 Mar 12 13:58 mongod.lock
drwxr-xr-x 2 oracle oinstall 4096 Mar 12 13:58 journal
rw-rr- 1 oracle oinstall 1912 Mar 12 14:05 loging_logs_shard2.log

Again here is what I did today:
oracle@dw07orl57 > mongo
MongoDB shell version: 2.0.3
connecting to: test
mongos> use admin
switched to db admin
mongos> db.printShardingStatus();
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard0000", "draining" : true, "host" : "dw07orl57:27018" } { "_id" : "shard0001", "draining" : true, "host" : "dw07orl57:27020" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "gis_db", "partitioned" : false, "primary" : "shard0000" }

mongos> use gis_db
switched to db gis_db
mongos> db.dropDatabase()

{ "assertion" : "collection's metadata is undergoing changes. Please try again.", "assertionCode" : 13331, "errmsg" : "db assertion failure", "ok" : 0 } mongos> show dbs config 0.1875GB gis_db 1.953125GB mongos> show dbs config 0.1875GB gis_db 1.953125GB mongos> use gis_db switched to db gis_db mongos> db.dropDatabase() { "dropped" : "gis_db", "ok" : 1 }

mongos> show dbs
config 0.1875GB
gis_db 1.953125GB

Comment by Eliot Horowitz (Inactive) [ 11/Mar/12 ]

That means a migration or split is happening.
If you try again, it should work.

Generated at Thu Feb 08 03:08:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.