We are running our application on MongoHQ's large replicaset plan with two nodes. One of them is currently working while the other one is unreachable. Looks like Mongoid doesn't know how to deal with this because after a long wait for the connection to go through it just crashes on boot.
Here's what our config looks like:
Unable to find source-code formatter for language: yml. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
production:
<<: *defaults
hosts:
- - polarbear.member1.mongohq.com
- 10009
- - polarbear.member0.mongohq.com
- 10009
username: xxx
password: xxx
database: xxx
read_secondary: true
First some Rails boot time warnings that we've gotten used to seeing for the past few months:
Initiating a ReplSetConnection with seeds passed as individual [host, port] array arguments is deprecated.
Please specify hosts as an array of 'host:port' strings; the old format will be removed in v2.0
:read_secondary options has now been deprecated and will be removed in driver v2.0. Use the :read option instead.
Then the following error:
/Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/tcp_socket.rb:64:in `select': execution expired (Timeout::Error) from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/tcp_socket.rb:64:in `read' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/networking.rb:338:in `receive_data' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/networking.rb:323:in `receive_message_on_socket' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/networking.rb:188:in `receive_header' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/networking.rb:175:in `receive' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/networking.rb:139:in `receive_message' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/cursor.rb:469:in `block in send_initial_query' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/logging.rb:36:in `instrument' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/cursor.rb:467:in `send_initial_query' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/cursor.rb:458:in `refresh' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/cursor.rb:128:in `next' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/db.rb:511:in `command' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/node.rb:83:in `set_config' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/pool_manager.rb:285:in `block in get_valid_seed_node' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/pool_manager.rb:281:in `each' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/pool_manager.rb:281:in `get_valid_seed_node' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/pool_manager.rb:166:in `connect_to_members' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/util/pool_manager.rb:30:in `connect' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/repl_set_connection.rb:155:in `block in connect' from <internal:prelude>:10:in `synchronize' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/repl_set_connection.rb:147:in `connect' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/connection.rb:589:in `setup' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/repl_set_connection.rb:515:in `setup' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongo-1.6.2/lib/mongo/repl_set_connection.rb:132:in `initialize' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config/replset_database.rb:22:in `new' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config/replset_database.rb:22:in `configure' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config.rb:288:in `configure_databases' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config.rb:111:in `from_hash' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config.rb:126:in `block in load!' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config.rb:125:in `tap' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/config.rb:125:in `load!' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid.rb:148:in `load!' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/mongoid-2.4.9/lib/mongoid/railtie.rb:84:in `block in <class:Railtie>' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/initializable.rb:30:in `instance_exec' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/initializable.rb:30:in `run' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/initializable.rb:55:in `block in run_initializers' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/initializable.rb:54:in `each' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/initializable.rb:54:in `run_initializers' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/application.rb:136:in `initialize!' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/railtie/configurable.rb:30:in `method_missing' from /Users/druiden/development/rails/streamio/config/environment.rb:8:in `block in <top (required)>' from /Users/druiden/development/rails/streamio/config/environment.rb:7:in `<top (required)>' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/activesupport-3.2.3/lib/active_support/dependencies.rb:251:in `require' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/activesupport-3.2.3/lib/active_support/dependencies.rb:251:in `block in require' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/activesupport-3.2.3/lib/active_support/dependencies.rb:236:in `load_dependency' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/activesupport-3.2.3/lib/active_support/dependencies.rb:251:in `require' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/application.rb:103:in `require_environment!' from /Users/druiden/.rvm/gems/ruby-1.9.3-p194@streamio/gems/railties-3.2.3/lib/rails/commands.rb:40:in `<top (required)>' from script/rails:6:in `require' from script/rails:6:in `<main>'
We temporarily solved the problem by configuring Mongoid to use only polarbear.member0... which is the working node but obviously defeating the purpose of replicasets in the process.
Let me know if you need more info.