Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22868

Production Mongo router fails to start

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 2.4.6
    • Sharding
    • None

    Description

      Hello,
      I currently support a production cluster of MongoDB sharded nodes in Gracenote as follows:

      • 3 shards, each has a primary and 2 replicas,
      • 3 config servers and
      • 3 router services on
      • all of them on MongoDB version 2.4.6
      • all of them in the same data center.

      Today our Operations group performed a rolling restart of all servers in relation to an emergency Linux patching. All servers and services came back fine, except the mongo router services on two servers, which report the errors shown below. The third router service however started fine. All related applications were modified to use only the running mongo router and not try to use the ones currently down.

      Can you please help help me bring the other two router services up?

      Here is the log file for the router service on dms-cr2-mongo-1-3:

      service mongodb_router start

      Thu Feb 25 22:16:45.991 [mongosMain] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
      Thu Feb 25 22:16:45.991 [mongosMain] options: { config: "/etc/mongodb_router.conf", configdb: "dms-cr2-mongo1-3.globix-sc.gracenote.com:27019,dms-cr2-mongo2-3.globix-sc.gracenote.com:27019,dms-cr2-mongo3-3.globix-sc.gracenote.com:27019", keyFile: "/data/mongokey/mongokey", logpath: "/data/router/log/mongodb.log", port: 27017 }
      Thu Feb 25 22:16:46.260 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 22:16:46.472 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 22:16:46.685 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 22:16:46.896 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 22:16:46.896 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      chunks: "3e093ea58d367d48df9955c4c5a83da7"    chunks: "ba2046891c22ceb79dfc1dbceaa37b2a"
      databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"    databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
      Thu Feb 25 22:16:46.896 [mongosMain] configServer connection startup check failed
      

      And for dms-cr2-mongo-2-3:

      service mongodb_router start

      Thu Feb 25 19:25:48.430 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
      Thu Feb 25 19:25:48.496 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 19:25:48.565 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
      Thu Feb 25 19:25:48.631 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 19:25:48.700 [mongosMain] warning:  couldn't check dbhash on config server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019 :: caused by :: 11002 socket exception [CONNECT_ERROR] server [dms-cr2-mongo2-3.globix-sc.gracenote.com:27019] mongos connectionpool error: couldn't connect to server dms-cr2-mongo2-3.globix-sc.gracenote.com:27019
      Thu Feb 25 19:25:48.767 [mongosMain] warning: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      Thu Feb 25 19:25:48.767 [mongosMain] ERROR: could not verify that config servers are in sync :: caused by :: config servers dms-cr2-mongo1-3.globix-sc.gracenote.com:27019 and dms-cr2-mongo3-3.globix-sc.gracenote.com:27019 differ
      chunks: "8f415e0cf812368e29512078431ca607"    chunks: "420016fbdd35884e2ffac54fd6bf3f95"
      databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"    databases: "00debde3ffe02a65dbe4bcbaeacb1fd4"
      Thu Feb 25 19:25:48.767 [mongosMain] configServer connection startup check failed
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            sdimitrova@gracenote.com Stefka Dimitrova
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: