Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Critical - P2 Critical - P2
    • Resolution: Works as Designed
    • Affects Version/s: 1.5.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
      ruby 1.9.2
      mongo / bson_ext 1.5.2
      heroku
      mongohq, 'gravity', replica set, 2 instances
    • Operating System:
      ALL
    • # Replies:
      7
    • Last comment by Customer:
      true

      Description

      coming from /app/.bundle/gems/ruby/1.9.1/gems/mongo-1.5.2/lib/mongo/util/pool.rb:72

        Activity

        Hide
        Kyle Banker
        added a comment -

        Do you have more of a stack trace for that?
        What do your replica set connection options look like?

        Show
        Kyle Banker
        added a comment - Do you have more of a stack trace for that? What do your replica set connection options look like?
        Hide
        Daniel Doubrovkine
        added a comment -

        I am working on trying to get a longer stack trace - pretty hard on heroku.

        This is my mongoid.yml

        defaults: &defaults
        allow_dynamic_fields: true
        parameterize_keys: true
        persist_in_safe_mode: true
        raise_not_found_error: true
        reconnect_time: 3
        use_object_ids: true

        production:
        <<: *defaults
        hosts: <%= ENV['MONGOHQ_HOST_LIST'] %>
        read_secondary: true
        database: <%= ENV['MONGOHQ_DATABASE'] %>
        username: <%= ENV['MONGOHQ_USER'] %>
        password: <%= ENV['MONGOHQ_PASSWD'] %>

        Show
        Daniel Doubrovkine
        added a comment - I am working on trying to get a longer stack trace - pretty hard on heroku. This is my mongoid.yml defaults: &defaults allow_dynamic_fields: true parameterize_keys: true persist_in_safe_mode: true raise_not_found_error: true reconnect_time: 3 use_object_ids: true production: <<: *defaults hosts: <%= ENV ['MONGOHQ_HOST_LIST'] %> read_secondary: true database: <%= ENV ['MONGOHQ_DATABASE'] %> username: <%= ENV ['MONGOHQ_USER'] %> password: <%= ENV ['MONGOHQ_PASSWD'] %>
        Hide
        Kyle Banker
        added a comment -

        Do you have any idea what triggered this? Was there a replica set connection failure? Are you doing anything out of the ordinary with Mongoid?

        Show
        Kyle Banker
        added a comment - Do you have any idea what triggered this? Was there a replica set connection failure? Are you doing anything out of the ordinary with Mongoid?
        Hide
        Daniel Doubrovkine
        added a comment - - edited

        Nothing out of the ordinary. It's reproducible right now in prod. We have happy hour starting in 30 if you want to stop by GA - it's a block away

        Note that 1.5.2 is brand new for us. We upgraded in emergency because we were hitting https://jira.mongodb.org/browse/RUBY-274 and that looked better than a dead env.

        Show
        Daniel Doubrovkine
        added a comment - - edited Nothing out of the ordinary. It's reproducible right now in prod. We have happy hour starting in 30 if you want to stop by GA - it's a block away Note that 1.5.2 is brand new for us. We upgraded in emergency because we were hitting https://jira.mongodb.org/browse/RUBY-274 and that looked better than a dead env.
        Hide
        Kyle Banker
        added a comment -

        Thanks for the offer.

        Really need more data on this, though. The best thing would be a stack trace. How often is this happening? How are you reproducing it?

        Show
        Kyle Banker
        added a comment - Thanks for the offer. Really need more data on this, though. The best thing would be a stack trace. How often is this happening? How are you reproducing it?
        Hide
        Daniel Doubrovkine
        added a comment -

        You can close this I think. This is what's happening.

        We found an infinite loop in an object relationship. That basically does x.y, then y.z, then z.x where x, y and z are mongoid objects. That causes the mongo 1.3.1 driver to end up in the state described in https://jira.mongodb.org/browse/RUBY-274 and kill our service without anything that points to the root problem, while 1.5.2 is much better and causes this error only with at least an indication of the top of the stack.

        With a local rails server I get a crash that brings down the server and a cryptic Illegal instruction: 4 on the console.
        With Heroku your dyno throws an exception and heroku console shows ! Internal server error.

        Show
        Daniel Doubrovkine
        added a comment - You can close this I think. This is what's happening. We found an infinite loop in an object relationship. That basically does x.y, then y.z, then z.x where x, y and z are mongoid objects. That causes the mongo 1.3.1 driver to end up in the state described in https://jira.mongodb.org/browse/RUBY-274 and kill our service without anything that points to the root problem, while 1.5.2 is much better and causes this error only with at least an indication of the top of the stack. With a local rails server I get a crash that brings down the server and a cryptic Illegal instruction: 4 on the console. With Heroku your dyno throws an exception and heroku console shows ! Internal server error.
        Hide
        Kyle Banker
        added a comment -

        Thanks, Daniel. I'm wondering if the gem could be better about handling this kind of situation, though I doubt it. If you can think of any better behavior, do let me know.

        Kyle

        Show
        Kyle Banker
        added a comment - Thanks, Daniel. I'm wondering if the gem could be better about handling this kind of situation, though I doubt it. If you can think of any better behavior, do let me know. Kyle

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Days since reply:
              2 years, 17 weeks, 2 days ago
              Date of 1st Reply: