[SERVER-22868] Production Mongo router fails to start Created: 26/Feb/16  Updated: 26/Feb/16  Resolved: 26/Feb/16

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Stefka Dimitrova Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Hello,
I currently support a production cluster of MongoDB sharded nodes in Gracenote as follows:

  • 3 shards, each has a primary and 2 replicas,
  • 3 config servers and
  • 3 router services on
  • all of them on MongoDB version 2.4.6
  • all of them in the same data center.

Today our Operations group performed a rolling restart of all servers in relation to an emergency Linux patching. All servers and services came back fine, except the mongo router services on two servers, which report the errors shown below. The third router service however started fine. All related applications were modified to use only the running mongo router and not try to use the ones currently down.

Can you please help help me bring the other two router services up?

Here is the log file for the router service on dms-cr2-mongo-1-3:

service mongodb_router start

Thu Feb 25 22:16:45.991 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Feb 25 22:16:45.991 [mongosMain] options: { config: "/etc/mongodb_router.conf", configdb: "dms-cr2-mongo1-3.globix-sc.gracenote.com:27019,dms-cr2-mongo2-3.globix-sc.gracenote.com:27019,dms-cr2-mongo3-3.globix-sc.gracenote.com:27019", keyFile: "/data/mongokey/mongokey", logpath: "/data/router/log/mongodb.log", port: 27017 }
Thu Feb 25 22:16:46.260 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 22:16:46.472 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 22:16:46.685 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 22:16:46.896 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 22:16:46.896 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
chunks: "3e093ea58d367d48df9955c4c5a83da7"    chunks: "ba2046891c22ceb79dfc1dbceaa37b2a"
databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"    databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
Thu Feb 25 22:16:46.896 [mongosMain] configServer connection startup check failed

And for dms-cr2-mongo-2-3:

service mongodb_router start

Thu Feb 25 19:25:48.430 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
Thu Feb 25 19:25:48.496 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 19:25:48.565 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
Thu Feb 25 19:25:48.631 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 19:25:48.700 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
Thu Feb 25 19:25:48.767 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
Thu Feb 25 19:25:48.767 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
chunks: "8f415e0cf812368e29512078431ca607"    chunks: "420016fbdd35884e2ffac54fd6bf3f95"
databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"    databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
Thu Feb 25 19:25:48.767 [mongosMain] configServer connection startup check failed



 Comments   
Comment by Ramon Fernandez Marina [ 26/Feb/16 ]

sdimitrova@gracenote.com, please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group.

Also note that 2.4.6 is very old at this stage, so if this behavior was caused by a bug it would not be fixed in 2.4, so I'd strongly recommend you plan an upgrade to a more recent version as soon as possible. MongoDB 3.2.3 is the latest one at the time of this writing, and it includes support for replica sets config servers, which seems you could benefit from.

See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 04:01:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.