|
Hello,
I currently support a production cluster of MongoDB sharded nodes in Gracenote as follows:
- 3 shards, each has a primary and 2 replicas,
- 3 config servers and
- 3 router services on
- all of them on MongoDB version 2.4.6
- all of them in the same data center.
Today our Operations group performed a rolling restart of all servers in relation to an emergency Linux patching. All servers and services came back fine, except the mongo router services on two servers, which report the errors shown below. The third router service however started fine. All related applications were modified to use only the running mongo router and not try to use the ones currently down.
Can you please help help me bring the other two router services up?
Here is the log file for the router service on dms-cr2-mongo-1-3:
service mongodb_router start
Thu Feb 25 22:16:45.991 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
|
Thu Feb 25 22:16:45.991 [mongosMain] options: { config: "/etc/mongodb_router.conf", configdb: "dms-cr2-mongo1-3.globix-sc.gracenote.com:27019,dms-cr2-mongo2-3.globix-sc.gracenote.com:27019,dms-cr2-mongo3-3.globix-sc.gracenote.com:27019", keyFile: "/data/mongokey/mongokey", logpath: "/data/router/log/mongodb.log", port: 27017 }
|
Thu Feb 25 22:16:46.260 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 22:16:46.472 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 22:16:46.685 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 22:16:46.896 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 22:16:46.896 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
chunks: "3e093ea58d367d48df9955c4c5a83da7" chunks: "ba2046891c22ceb79dfc1dbceaa37b2a"
|
databases: "00debde3ffe02a65dbe4bcbaeacb1fd4" databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
|
Thu Feb 25 22:16:46.896 [mongosMain] configServer connection startup check failed
|
And for dms-cr2-mongo-2-3:
service mongodb_router start
Thu Feb 25 19:25:48.430 [mongosMain] warning: couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
|
Thu Feb 25 19:25:48.496 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 19:25:48.565 [mongosMain] warning: couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
|
Thu Feb 25 19:25:48.631 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 19:25:48.700 [mongosMain] warning: couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
|
Thu Feb 25 19:25:48.767 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
Thu Feb 25 19:25:48.767 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
|
chunks: "8f415e0cf812368e29512078431ca607" chunks: "420016fbdd35884e2ffac54fd6bf3f95"
|
databases: "00debde3ffe02a65dbe4bcbaeacb1fd4" databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
|
Thu Feb 25 19:25:48.767 [mongosMain] configServer connection startup check failed
|
|