[SERVER-5630] mongos unable to start give error "differconfig servers not in sync! config servers ". Created: 17/Apr/12  Updated: 26/Sep/16  Resolved: 01/Jun/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: jitendra Assignee: Spencer Brody (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

oracle linux


Issue Links:
Related
related to SERVER-5681 unable to start mongod give exception... Closed
is related to SERVER-5380 unable to start mongod give exception... Closed
is related to SERVER-5598 Run command on mongos> db.stats() g... Closed
Operating System: Linux
Participants:

 Description   

hi,

we setup mongo servers on two machine. All mongod run with journal option.

machine_1 : 4 mongod, 2 config server, 1 mongos.
machine_2 : 4 mongod, 1 config server.

I reboot machine_1. start all mongods and config server after this start mongos.
mongos is unable to start. error logs are below:

          • SERVER RESTARTED *****

Tue Apr 17 18:49:17 /usr/local/ct/depend/mongo/bin/mongos db version v2.0.3-rc0, pdfile version 4.5 starting (--help for usage)
Tue Apr 17 18:49:17 git version: 643c3a25c7fa272a3ff343a7ed653f0cef17f60f
Tue Apr 17 18:49:17 build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Apr 17 18:49:17 config string : 192.168.129.112:30000,192.168.129.112:32000,192.168.129.93:37000
Tue Apr 17 18:49:17 creating new connection to:192.168.129.112:30000
Tue Apr 17 18:49:17 BackgroundJob starting: ConnectBG
Tue Apr 17 18:49:17 connected connection!
Tue Apr 17 18:49:17 creating new connection to:192.168.129.112:32000
Tue Apr 17 18:49:17 BackgroundJob starting: ConnectBG
Tue Apr 17 18:49:17 connected connection!
Tue Apr 17 18:49:17 creating new connection to:192.168.129.93:37000
Tue Apr 17 18:49:17 BackgroundJob starting: ConnectBG
Tue Apr 17 18:49:17 connected connection!
Tue Apr 17 18:49:17 ERROR: config servers 192.168.129.112:30000 and 192.168.129.93:37000 differconfig servers 192.168.129.112:30000 and 192.168.129.93:37000 differconfig servers 192.168.129.112:30000 and 192.168.129.93:37000 differconfig servers 192.168.129.112:30000 and 192.168.129.93:37000 differconfig servers not in sync! config servers 192.168.129.112:30000 and 192.168.129.93:37000 differ
chunks: "25e79eeea13d87d4be49781249e1db44" chunks: "0b8d1c4a7d18645610be67e9ebd721cf"
databases: "6df2443ff88a2726f746ebf6c547a909" databases: "6df2443ff88a2726f746ebf6c547a909"
configServer startup check failed
CursorCache at shutdown - sharded: 0 passthrough: 0

kindly help me how to start mongos.

Thanks in advance
jitendra verma.



 Comments   
Comment by Jean Nascimento [ 07/Jan/13 ]

Worked, thanks for the tip!

Comment by Spencer Brody (Inactive) [ 23/Apr/12 ]

We generally don't recommend automating any change that involves messing with config data, as that is very risky and if done incorrectly can leave your cluster unusable (if the config data gets messed up the mongoses won't know where to route requests for data). It should be very rare for your config servers to get out of sync. We are planning to create scripts to make this process easier in the future (SERVER-5414) but they will be very conservative and thoroughly tested.

That said, every step of the process I outlined above should be possible to script fairly easily - just be sure to test any such script thoroughly and use at your own risk.

Comment by jitendra [ 22/Apr/12 ]

hi
how can be automate this solution in running mongo setup.

Thanks in advance
jitendra verma.

Comment by Spencer Brody (Inactive) [ 17/Apr/12 ]

It looks like your config servers got out of sync. There was probably a migration or chunk split in progress when you shut down the machine.

Your best bet is to connect to each of the config servers directly and query the changelog collection in the config database to find which of them has the entry with the latest timestamp. Once you identify which config server has the most up-to-date copy of the data you should restore its data onto the other 2 config servers. Once the servers are brought back into sync, the mongos should be able to start up properly.

Generated at Thu Feb 08 03:09:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.