Details

    • Type: Bug Bug
    • Status: Open Open
    • Priority: Major - P3 Major - P3
    • Resolution: Unresolved
    • Affects Version/s: 1.6.5
    • Fix Version/s: Planning Bucket C
    • Component/s: Networking
    • Labels:
      None
    • Environment:
      Ubuntu 64 bit 10.10 using 10gen stable package
    • Backport:
      No
    • Operating System:
      Linux
    • # Replies:
      7
    • Last comment by Customer:
      true

      Description

      The socket listen backlog is not configurable. I have a situation where a large number of worker processes receive work notification from outside of Mongo and then all simultaneously connect to mongo get more details. They overwhelm the listen backlog and consequently some get connection refused. Most daemons let you configure the listen backlog size away from the normal OS default of 5.

        Activity

        Hide
        Eliot Horowitz
        added a comment -

        We could just raise it to 128.
        I think that's what mysql does for example.
        Thoughts?

        Show
        Eliot Horowitz
        added a comment - We could just raise it to 128. I think that's what mysql does for example. Thoughts?
        Hide
        Roger Binns
        added a comment -

        128 would certainly have fixed the issue for me. However it is traditional that the number is configurable (command line flag) since it is unlikely one number fits all circumstances.

        Show
        Roger Binns
        added a comment - 128 would certainly have fixed the issue for me. However it is traditional that the number is configurable (command line flag) since it is unlikely one number fits all circumstances.
        Hide
        John Zwinck
        added a comment -

        Isn't it already set to 128? I did the archaeology and found this which seems to be the original code calling listen(2) four years ago, with a backlog of 128: https://github.com/mongodb/mongo/commit/5ab9ae9d47df7fb32e6dd95b404e783020281218#grid/message.cpp

        So, is this bug valid? Does it refer to something I'm not seeing?

        Show
        John Zwinck
        added a comment - Isn't it already set to 128? I did the archaeology and found this which seems to be the original code calling listen(2) four years ago, with a backlog of 128: https://github.com/mongodb/mongo/commit/5ab9ae9d47df7fb32e6dd95b404e783020281218#grid/message.cpp So, is this bug valid? Does it refer to something I'm not seeing?
        Hide
        Eliot Horowitz
        added a comment -

        The reason its open is to figure out if we should make it configurable.
        I'm leaning towards no - but not 100% sure.

        Show
        Eliot Horowitz
        added a comment - The reason its open is to figure out if we should make it configurable. I'm leaning towards no - but not 100% sure.
        Hide
        Roger Binns
        added a comment -

        The limit is still a problem for me. Once more worker machines come online I'll be having the listen length issue again.

        All of my worker processes (not threads due to Python GIL) talk to MongoDB. I end up with over 1,000 connections to MongoDB and all of those happen at startup virtually instantaneously (lots of forks).

        I'm rather sceptical that one size will fit everyone hence a need for it to be configurable. There is a reason why virtually every UNIX daemon that will serve non-trivial loads has it as a configurable, and rarely use the OS default of 5.

        As an example something else I use is gearman which takes --backlog and a default of 32. MySql takes --back_log and I don't know what the default is. I don't run postgres so I can't tell what it does, but I did find a CVS commit from last year making the listen backlog configurable.

        So it is evident that other databases and similar system components had the need to make the value configurable. What makes using MongoDB so special that its default is the perfect value for everyone?

        Show
        Roger Binns
        added a comment - The limit is still a problem for me. Once more worker machines come online I'll be having the listen length issue again. All of my worker processes (not threads due to Python GIL) talk to MongoDB. I end up with over 1,000 connections to MongoDB and all of those happen at startup virtually instantaneously (lots of forks). I'm rather sceptical that one size will fit everyone hence a need for it to be configurable. There is a reason why virtually every UNIX daemon that will serve non-trivial loads has it as a configurable, and rarely use the OS default of 5. As an example something else I use is gearman which takes --backlog and a default of 32. MySql takes --back_log and I don't know what the default is. I don't run postgres so I can't tell what it does, but I did find a CVS commit from last year making the listen backlog configurable. So it is evident that other databases and similar system components had the need to make the value configurable. What makes using MongoDB so special that its default is the perfect value for everyone?
        Hide
        Eliot Horowitz
        added a comment -

        The goal is to have as few options as needed.
        So if there is an intelligent way to not make it configurable, then it would be ideal
        If that's not possible - then we'll make it configurable.

        Show
        Eliot Horowitz
        added a comment - The goal is to have as few options as needed. So if there is an intelligent way to not make it configurable, then it would be ideal If that's not possible - then we'll make it configurable.
        Hide
        Roger Binns
        added a comment -

        To my knowledge there is no way to make it adaptive (ie not need configuration). You can look at the rate of incoming connections, but you can't tell when a connection initially arrived (ie how long it was in the listen queue before accept was called). You also get no notification whatsoever about the connection requests that were rejected because the queue was full.

        (You could in theory see how long something was in the listen queue if you also had a network tap and looked at the TCP headers and understood the operating system behaviour with regard to TCP timestamping. However this is orders of magnitude more code and considerably more fragile than an integer option.)

        Show
        Roger Binns
        added a comment - To my knowledge there is no way to make it adaptive (ie not need configuration). You can look at the rate of incoming connections, but you can't tell when a connection initially arrived (ie how long it was in the listen queue before accept was called). You also get no notification whatsoever about the connection requests that were rejected because the queue was full. (You could in theory see how long something was in the listen queue if you also had a network tap and looked at the TCP headers and understood the operating system behaviour with regard to TCP timestamping. However this is orders of magnitude more code and considerably more fragile than an integer option.)

          People

          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Days since reply:
              2 years, 18 weeks, 2 days ago
              Date of 1st Reply: