Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17652

Cannot start mongod due to "sockets higher than 1023 not being supported"

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.0.2, 3.1.1
    • Component/s: Networking, Storage
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:
    • Sprint:
      RPL 1 04/03/15

      Description

      Issue Status as of Apr 02, 2015

      ISSUE SUMMARY

      If a mongod process opens too many file descriptors early in the initialization process, the process may fail to start if later parts of the initialization process require the server to allocate a socket. For instance, this may arise when the mongod runs with the httpinterface option, and the storage engine needs to open a large number of file descriptors, as happens during recovery following an unclean shutdown.

      If you observe the following error, this may indicate that you've encountered this issue.

      sockets higher than 1023 not supported
      

      WORKAROUNDS

      Disable the httpinterface until mongod can effectively start with a smaller number of file descriptors.

      AFFECTED VERSIONS
      3.0.0+

      FIX VERSION
      The fix is included in the 3.0.2 production release.

      Original description

      There are several community reports of users who have upgraded from 3.0.0 to 3.0.1 and are unable to restart mongodb, with an error message about sockets higher than 1023 not being supported, eg:

      2015-03-18T12:40:38.204+0800 E NETWORK  [initandlisten] socket 31741 is higher than 1023; not supported
      2015-03-18T12:40:38.204+0800 I -        [initandlisten] Fatal Assertion 18823
      2015-03-18T12:40:38.320+0800 I CONTROL  [initandlisten]
       0xf69089 0xf09881 0xeee401 0xc78dd9 0xacb2a1 0x7f13eb 0x7effec19f76d 0x822479
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"B69089"},{"b":"400000","o":"B09881"},{"b":"400000","o":"AEE401"},{"b":"400000","o":"878DD9"},{"b":"400000","o":"6CB2A1"},{"b":"400000","o":"3F13EB"},{"b":"7EFFEC17E000","o":"2176D"},{"b":"400000","o":"422479"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10f00cd27737fbcd951032248b5952", "uname" : { "sysname" : "Linux", "release" : "3.2.0-64-generic", "version" : "#97-Ubuntu SMP Wed Jun 4 22:04:21 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7D0BB0BFBF6B9A69384E0AB86648732D9C5BA8" }, { "b" : "7FFF6BEFF000", "elfType" : 3, "buildId" : "DC13108D2C20205E89646A22C627FAE0FE5137C4" }, { "b" : "7EFFED794000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7EFFED536000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "637E94470A47FC27ECFFFE89F964D001BFC86673" }, { "b" : "7EFFED15B000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2A5A25CDD987E6FD5BAEE0B53CC47CE5D824E183" }, { "b" : "7EFFECF53000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7EFFECD4F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7EFFECA4F000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7EFFEC753000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7EFFEC53D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7EFFEC17E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "57B3EB388FB6FE036D1E708259DACA7FCE990462" }, { "b" : "7EFFED9B1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" }, { "b" : "7EFFEBF67000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf69089]
       mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf09881]
       mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeee401]
       mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl8shutdownEv+0x1F9) [0xc78dd9]
       mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x61) [0xacb2a1]
       mongod(main+0x13B) [0x7f13eb]
       libc.so.6(__libc_start_main+0xED) [0x7effec19f76d]
       mongod(+0x422479) [0x822479]
      -----  END BACKTRACE  -----
      2015-03-18T12:40:38.320+0800 I -        [initandlisten]
       
      ***aborting after fassert() failure
      

      Some reports:

        Issue Links

          Activity

          Hide
          ramon.fernandez Ramon Fernandez added a comment -

          leonty, WiredTiger should only need to open lots of files when recovery is needed, so on the next restart the server should come up as recovery should not be needed. Are you saying that restarting after seeing the error above doesn't work?

          Show
          ramon.fernandez Ramon Fernandez added a comment - leonty , WiredTiger should only need to open lots of files when recovery is needed, so on the next restart the server should come up as recovery should not be needed. Are you saying that restarting after seeing the error above doesn't work?
          Hide
          greg_white Greg White added a comment -

          I'm not able to restart one of my slaves once it gets in this state. I get the same error every time. Is there no workaround?

          Show
          greg_white Greg White added a comment - I'm not able to restart one of my slaves once it gets in this state. I get the same error every time. Is there no workaround?
          Hide
          leonty leonty added a comment -

          Ramon, exactly. Restarts don't help. I ensured that the the upstart script sets needed limits. I also rised up limits globally for the "mongodb" user in /etc/security/limits.conf. Nothing helped. I had to downgrade mongodb to 3.0.0.

          Show
          leonty leonty added a comment - Ramon, exactly. Restarts don't help. I ensured that the the upstart script sets needed limits. I also rised up limits globally for the "mongodb" user in /etc/security/limits.conf. Nothing helped. I had to downgrade mongodb to 3.0.0.
          Hide
          leonty leonty added a comment -

          3.0.2 is finally released. Thank you!

          Show
          leonty leonty added a comment - 3.0.2 is finally released. Thank you!
          Hide
          ramon.fernandez Ramon Fernandez added a comment -

          Hi leonty, apologies for not getting back to you sooner. In the future you may want to subscribe to mongodb-announce@googlegroups.com where we announce our releases. In this case you would have been able to download and test 3.0.2-rc0 (which had the same bits as 3.0.2) to confirm that this issue was actually addressed as early as April 2nd.

          Show
          ramon.fernandez Ramon Fernandez added a comment - Hi leonty , apologies for not getting back to you sooner. In the future you may want to subscribe to mongodb-announce@googlegroups.com where we announce our releases. In this case you would have been able to download and test 3.0.2-rc0 (which had the same bits as 3.0.2) to confirm that this issue was actually addressed as early as April 2nd.

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              29 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                  Agile