[SERVER-12268] Missing collections when performing "show collections" on mongos Created: 07/Jan/14  Updated: 10/Dec/14  Resolved: 20/May/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Simon Schneider Assignee: Thomas Rueckstiess
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Debian Wheezy


Operating System: Linux
Participants:

 Description   

I'm missing collections when performing "show collections" on a mongos. My Setup is a sharded Cluster with 3 config serves and currently 7 shards.

I checked one collection more closely and can see this collection in sh.status(). I can also see the collection on all 3 config servers by doing db.collections.find(). Also a db.collection.find() on the mongos provides a correct answer.

Another effect is that chunk balancer is failing with

{ ok: 0.0, errmsg: "ns not found, should be impossible" }

 Comments   
Comment by Thomas Rueckstiess [ 20/May/14 ]

Hi Anthony,

Glad to hear this is no longer an issue for you. As for the "out of memory" failure: There is already a ticket open to request a more graceful failure if mongod runs out of memory: SERVER-13719. You can watch this ticket for progress on the improvement.

I'll resolve this issue now as we can't reproduce the problem.

Kind regards,
Thomas

Comment by Anthony Leung [ 02/May/14 ]

Hi, I moved it to another server so it's not an issue for me anymore.

Just upgraded this problem server now to see if it would fix it, noticed
the following in the logs relating to another database in the same server.

Fri May 2 14:07:45.768 [conn1] ERROR: mmap() failed for
/home/ubuntu/db/mongodb/crysnet.4 len:268435456 errno:12 Cannot allocate
memory
Fri May 2 14:07:45.768 [conn1] ERROR: mmap failed with out of memory. You
are using a 32-bit build and probably need to upgrade to 64
 
Couldn't drop it to see if that would fix it.
switched to db crysnet
{
        "ok" : 0,
        "errmsg" : "_extentManager.init failed: InternalError
DataFile::openExisting - mmf.open failed",
        "code" : 16966
}

How about stopping the mongodb instance when the user hits the limit 32
bit, and not allowing it to start?

Anthony

On Thu, May 1, 2014 at 11:50 PM, Thomas Rueckstiess (JIRA) <jira@mongodb.org

Comment by Thomas Rueckstiess [ 01/May/14 ]

Hi Anthony,

I just wanted to see if you upgraded to 2.6 already and still experience this issue? If so, we'd like to continue investigating this problem. Please let me know.

Thanks,
Thomas

Comment by Anthony Leung [ 13/Apr/14 ]

32 bit server. I may have copied files, but not while the server was actually being used. Not sharded.

> db.version()
2.4.2-rc0
> show tables
cifs
system.indexes
urls
> db.getCollectionNames()
[ "cifs", "system.indexes", "urls" ]
> db.system.namespaces.find().forEach( printjson );

{ "name" : "icedragon.system.indexes" } { "name" : "icedragon.urls.$_id_" } { "name" : "icedragon.urls" } { "name" : "icedragon.cifs.$_id_" } { "name" : "icedragon.cifs" }

> db.content.findOne()
{
"_id" : ObjectId("5325d2ae0882ab50c5077553"),
"url" : "http://www.r.....

I'm about to upgrade to 2.6 anyway, but attempting to rename it:

> db.content.renameCollection('dontent')
{
"errmsg" : "exception: assertion src/mongo/db/namespace_details.cpp:926",
"code" : 0,
"ok" : 0
}
the rename works, but the collection is still missing from getCollectionNames.

There's something on stackoverflow from another user about this too.

Might just be 32 bit limit or something. Upgrading to 64 bit 2.6 anyhow, hope this helps.

Comment by Stennie Steneker (Inactive) [ 17/Mar/14 ]

Hi Simon,

Are you still experiencing this issue, or were you able to determine a cause?

The problems seemed to start after we updated MongoDB from Version 2.2.2 to 2.4.8 . Are there incompatibilities with old pymongo versions to MongoDB 2.4? We are using a pretty old version pymongo 1.7 from Debian Squeeze.

I'm not aware of any specific incompatibilities of pymongo 1.7 aside from lacking support for newer server features which weren't available when the driver was released, however there have certainly been numerous improvements in both MongoDB and the PyMongo driver in the ~4 years since PyMongo 1.7 was released. It would definitely be worth upgrading.

Regards,
Stephen

Comment by Simon Schneider [ 15/Jan/14 ]

I've seen that collections are missing only on primary shard configured for the database. I looked into system.namespaces collection directly on mongo shards and namespaces are not existent on primary shard.
The only way i can figure out that someone would have do a drop collection directly on mongo shard. But I hope this is not the reason.

The problems seemed to start after we updated MongoDB from Version 2.2.2 to 2.4.8 . Are there incompatibilities with old pymongo versions to MongoDB 2.4? We are using a pretty old version pymongo 1.7 from Debian Squeeze.

Comment by Eliot Horowitz (Inactive) [ 11/Jan/14 ]

When connected directly to the mongod shard, can you send the output of

use <correct db>
db.system.namespaces.find().forEach( printjson );

Comment by Simon Schneider [ 09/Jan/14 ]

It seems to me that this has something to do with the collection names. Collection names are as followed:
name_3-4digit_10*Hex_Date_5*Hex

So it looks something like this:
Products_1234_ABCDEF1234_20140109_123AB

We're now having more than one collection starting with "Products_1234_ABCDEF1234_*******_****" but have another date set and the last hex number is also different, But I'm only able to see one of those collection with "show collections". Can this be a problem with those long, similar collection names? I don't find any restriction but 123bytes for namespace, and including 8 characters for database name, im not exceeding this value.

I did also see, while connected directly to a mongodb server, that "show collections" give the same output, so this is no problem with mongos ( i thought so, because chunk balancing does also not work).

Generated at Thu Feb 08 03:28:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.