[SERVER-19280] messy database names in mongos Created: 03/Jul/15  Updated: 09/Jan/16  Resolved: 09/Jan/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.0.3, 3.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: DaixiShi Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File messy_code.png     PNG File messy_code2.png     File mongos1.rar     File mongos2.rar     PNG File sh1.png     PNG File sh4.png    
Operating System: ALL
Sprint: Sharding B (10/30/15)
Participants:

 Description   

I have a sharded cluster with 4 shards using MongoDB 3.0.3 and WiredTiger engine.
Recently, when I log in mongos and type `sh.status()`, I find many messy names of database shown in attachment.
But when I log in the primary shard of the messy databases specified in the `sh.status()`, I can't find those messy databases shown in attachment.
I'm sure I don't insert these messy databases in my cluster.



 Comments   
Comment by Ramon Fernandez Marina [ 09/Jan/16 ]

tottishi05, we haven't heard back from you for some time so I'm going to close this ticket. If this is still an issue for you please let us know so we can reopen the ticket.

Regards,
Ramón.

Comment by Sam Kleinman (Inactive) [ 11/Nov/15 ]

Hello,

Sorry for not getting back to you sooner. I think that there are a few things we can do to move this forward. First, we want to be absolutely sure that we can rule out an issue with your driver:

  1. Please be sure to use the latest release of the C# driver.
  2. Ensure that no other drivers or tools may be inadvertently creating databases outside of your driver.
  3. If this problem continues to happen, even after upgrading your cluster (see below,) it might be useful to see if you can reproduce the issue using a different driver or client (including the mongo shell) to make sure that it's not a problem in either C# or your application code.

Secondly, and perhaps more importantly it's very possible that you've run into an issue that the change included in SERVER-17723 would resolve.

Could you consider running your application in a test environment using 3.2.0-rc2?

Thanks for your time, and sorry again for the delay.

Cheers,
sam

Comment by DaixiShi [ 14/Jul/15 ]

shard04 with correct namespace

Comment by DaixiShi [ 14/Jul/15 ]

shard01 with messy collections

Comment by DaixiShi [ 14/Jul/15 ]

Thanks for Kaloian Manassiev.
It is unfortunate that not only mongos but also the shard is corrupted with messy namespaces as the new pictures attached.
Only the "shard01" has messy tables/collections.

Comment by Kaloian Manassiev [ 13/Jul/15 ]

Some notes on the progress of investigation:

In MongoDB 3.0 every request, which needs to check for the existence of a particular namespace, whether it is read or write, would create a database entry. This was fixed in 3.2 as part of SERVER-17723.

Since the databases with the messy names are only present on MongoS (i.e., the config metadata) but not on the shards themselves, this must either have been a read operation, or it was a write operation, which subsequently failed on MongoD.

Comment by DaixiShi [ 08/Jul/15 ]

Hi, Kaloian Manassiev.
I have uploaded the two logs of mongos process.
If you find the reason of my problem, please notice me.
Thanks very much

Comment by Kaloian Manassiev [ 07/Jul/15 ]

Thank you for the information, tottishi05.

Let me check around to see whether there are some known issues with the C# driver at this version.

As of this time I am working on the hypothesis that the driver somehow sent a wrong BSON and the command value was interpreted as a database name instead of command, so it should not be related to the combination of 3.0.3 and 3.0.4 in any way.

Did this issue happen over a recent time period and is there any possibility that you still have the mongos log files from around the time it happened? If you still have them, would you mind attaching them to the ticket - this may greatly help with debugging.

Note that if you are uploading logs and they may contact any sensitive information, send me a private message and I will give you a secure SCP server.

Thank you for your help.

Best regards,
-Kal.

Comment by DaixiShi [ 07/Jul/15 ]

Thanks for Kaloian Manassiev!
My sharded cluster is deployed on Cent OS 2.6.4.
I use the C# driver `MongoDB.Driver` and `MongoDB.Bson` of version 1.10.0.62.
With your lead, I check the version of all my mongod and mongos processes and find that:
1. all of the three mongod processes of config server are using 3.0.3
2. mongod process of shard 4 is using the 3.0.4 verion but all of the other mongods are using 3.0.4
3. One of the mongos process is using 3.0.3 with the other using 3.0.4

So, do you think my problem is associated with my mongodb version of 3.0.3 and 3.0.4??

Comment by Kaloian Manassiev [ 06/Jul/15 ]

Hi Daixi,

Thank you very much for reporting this issue.

From a first look, it appears like the messed up database names include partial BSON binary data from a command (for example, the two that begin with MediaDissector).

What driver are you using to connect to mongos? I am wondering whether the driver might be constructing an incorrect command namespace, which in turn was incorrectly interpreted by mongos as the name of the database instead of as a command action.

Also, did this happen after an upgrade or clean 3.0 install and are all of your instances (both mongos and mongod) at 3.0?

Thank you in advance.

-Kal.

Comment by DaixiShi [ 03/Jul/15 ]

I have two mongos process.
I log in them and get the same result.

Comment by Ramon Fernandez Marina [ 03/Jul/15 ]

tottishi05, do you observe the same behavior in all your mongos nodes? Can you please connect to all your mongos instances and run the same commands?

Generated at Thu Feb 08 03:50:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.