[SERVER-17652] Cannot start mongod due to "sockets higher than 1023 not being supported" Created: 19/Mar/15  Updated: 19/Sep/15  Resolved: 24/Mar/15

Status: Closed
Project: Core Server
Component/s: Networking, Storage
Affects Version/s: 3.0.1
Fix Version/s: 3.0.2, 3.1.1

Type: Bug Priority: Critical - P2
Reporter: Stennie Steneker (Inactive) Assignee: Eric Milkie
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-15389 Cannot start mongod when opening too ... Closed
is related to SERVER-17653 ERROR: socket XXX is higher than 1023... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: RPL 1 04/03/15
Participants:

 Description   
Issue Status as of Apr 02, 2015

ISSUE SUMMARY

If a mongod process opens too many file descriptors early in the initialization process, the process may fail to start if later parts of the initialization process require the server to allocate a socket. For instance, this may arise when the mongod runs with the httpinterface option, and the storage engine needs to open a large number of file descriptors, as happens during recovery following an unclean shutdown.

If you observe the following error, this may indicate that you've encountered this issue.

sockets higher than 1023 not supported

WORKAROUNDS

Disable the httpinterface until mongod can effectively start with a smaller number of file descriptors.

AFFECTED VERSIONS
3.0.0+

FIX VERSION
The fix is included in the 3.0.2 production release.

Original description

There are several community reports of users who have upgraded from 3.0.0 to 3.0.1 and are unable to restart mongodb, with an error message about sockets higher than 1023 not being supported, eg:

2015-03-18T12:40:38.204+0800 E NETWORK  [initandlisten] socket 31741 is higher than 1023; not supported
2015-03-18T12:40:38.204+0800 I -        [initandlisten] Fatal Assertion 18823
2015-03-18T12:40:38.320+0800 I CONTROL  [initandlisten]
 0xf69089 0xf09881 0xeee401 0xc78dd9 0xacb2a1 0x7f13eb 0x7effec19f76d 0x822479
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B69089"},{"b":"400000","o":"B09881"},{"b":"400000","o":"AEE401"},{"b":"400000","o":"878DD9"},{"b":"400000","o":"6CB2A1"},{"b":"400000","o":"3F13EB"},{"b":"7EFFEC17E000","o":"2176D"},{"b":"400000","o":"422479"}],"processInfo":{ "mongodbVersion" : "3.0.1", "gitVersion" : "534b5a3f9d10f00cd27737fbcd951032248b5952", "uname" : { "sysname" : "Linux", "release" : "3.2.0-64-generic", "version" : "#97-Ubuntu SMP Wed Jun 4 22:04:21 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "7F7D0BB0BFBF6B9A69384E0AB86648732D9C5BA8" }, { "b" : "7FFF6BEFF000", "elfType" : 3, "buildId" : "DC13108D2C20205E89646A22C627FAE0FE5137C4" }, { "b" : "7EFFED794000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7EFFED536000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "637E94470A47FC27ECFFFE89F964D001BFC86673" }, { "b" : "7EFFED15B000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2A5A25CDD987E6FD5BAEE0B53CC47CE5D824E183" }, { "b" : "7EFFECF53000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7EFFECD4F000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7EFFECA4F000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7EFFEC753000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7EFFEC53D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7EFFEC17E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "57B3EB388FB6FE036D1E708259DACA7FCE990462" }, { "b" : "7EFFED9B1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" }, { "b" : "7EFFEBF67000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf69089]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xf09881]
 mongod(_ZN5mongo13fassertFailedEi+0x61) [0xeee401]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl8shutdownEv+0x1F9) [0xc78dd9]
 mongod(_ZN5mongo11exitCleanlyENS_8ExitCodeE+0x61) [0xacb2a1]
 mongod(main+0x13B) [0x7f13eb]
 libc.so.6(__libc_start_main+0xED) [0x7effec19f76d]
 mongod(+0x422479) [0x822479]
-----  END BACKTRACE  -----
2015-03-18T12:40:38.320+0800 I -        [initandlisten]
 
***aborting after fassert() failure

Some reports:



 Comments   
Comment by Ramon Fernandez Marina [ 10/Apr/15 ]

Hi leonty, apologies for not getting back to you sooner. In the future you may want to subscribe to mongodb-announce@googlegroups.com where we announce our releases. In this case you would have been able to download and test 3.0.2-rc0 (which had the same bits as 3.0.2) to confirm that this issue was actually addressed as early as April 2nd.

Comment by leonty [ 10/Apr/15 ]

3.0.2 is finally released. Thank you!

Comment by leonty [ 30/Mar/15 ]

Ramon, exactly. Restarts don't help. I ensured that the the upstart script sets needed limits. I also rised up limits globally for the "mongodb" user in /etc/security/limits.conf. Nothing helped. I had to downgrade mongodb to 3.0.0.

Comment by Greg White [ 30/Mar/15 ]

I'm not able to restart one of my slaves once it gets in this state. I get the same error every time. Is there no workaround?

Comment by Ramon Fernandez Marina [ 30/Mar/15 ]

leonty, WiredTiger should only need to open lots of files when recovery is needed, so on the next restart the server should come up as recovery should not be needed. Are you saying that restarting after seeing the error above doesn't work?

Comment by leonty [ 30/Mar/15 ]

But what's the workaround for 3.0.1? I can't just start the database...

I have the following configuration:

2015-03-30T11:39:55.229+0300 I CONTROL  ***** SERVER RESTARTED *****
2015-03-30T11:39:55.234+0300 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),hazard_max=10000
2015-03-30T11:39:56.667+0300 I CONTROL  [initandlisten] MongoDB starting : pid=9564 port=27017 dbpath=/var/lib/mongodb 64-bit host=db2.btf.local
2015-03-30T11:39:56.667+0300 I CONTROL  [initandlisten]
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten]
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten]
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] db version v3.0.1
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] git version: 534b5a3f9d10f00cd27737fbcd951032248b5952
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] build info: Linux ip-10-167-176-91 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] allocator: tcmalloc
2015-03-30T11:39:56.668+0300 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", wiredTiger: { engineConfig: { configString: "hazard_max=10000" } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2015-03-30T11:39:56.782+0300 E NETWORK  [initandlisten] socket 1103 is higher than 1023; not supported
2015-03-30T11:39:56.782+0300 I CONTROL  [initandlisten] now exiting

Comment by Githook User [ 26/Mar/15 ]

Author:

{u'username': u'visualzhou', u'name': u'Siyuan Zhou', u'email': u'siyuan.zhou@mongodb.com'}

Message: SERVER-17652 Only run too_many_fds.js on 64-bit builds.

(cherry picked from commit 2c6ab1eaae38caeee322f0c9afc4ef973fb667ea)
Branch: v3.0
https://github.com/mongodb/mongo/commit/b82dcb7415ee4bcb949dacbb1a2347c85008961d

Comment by Githook User [ 26/Mar/15 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-17652 open sockets before initializing storage engine

(cherry picked from commit b9b1c95520824c377b4c800e6193d9855be1e14d)
Branch: v3.0
https://github.com/mongodb/mongo/commit/fac3991b11a8c026aaf02175b2b5a43e31c8d103

Comment by Ramon Fernandez Marina [ 20/Mar/15 ]

Hi igor,

as per the fixVersion we're targeting a 3.0.2 backport (the "backport approved" field went away).

Comment by Igor Canadi [ 20/Mar/15 ]

It would be awesome if you guys could backport this to v3.0 branch. RocksDB creates a lot of files and we've run into this issue during our testing.

Comment by Githook User [ 19/Mar/15 ]

Author:

{u'username': u'visualzhou', u'name': u'Siyuan Zhou', u'email': u'siyuan.zhou@mongodb.com'}

Message: SERVER-17652 Only run too_many_fds.js on 64-bit builds.
Branch: master
https://github.com/mongodb/mongo/commit/2c6ab1eaae38caeee322f0c9afc4ef973fb667ea

Comment by Githook User [ 19/Mar/15 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-17652 open sockets before initializing storage engine
Branch: master
https://github.com/mongodb/mongo/commit/b9b1c95520824c377b4c800e6193d9855be1e14d

Generated at Thu Feb 08 03:45:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.